Oct 30 05:29:57.575300 ip-10-0-138-200 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:29:58.009097 ip-10-0-138-200 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:29:58.009097 ip-10-0-138-200 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:29:58.009097 ip-10-0-138-200 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:29:58.009097 ip-10-0-138-200 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:29:58.013577 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013494 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:29:58.013577 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013577 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:29:58.013577 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013581 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013584 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013587 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013591 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013594 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013596 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013599 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013601 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013604 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013607 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013609 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013612 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013614 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013617 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013619 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013621 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013624 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013626 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013629 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013631 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:29:58.013677 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013634 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013637 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013640 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013642 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013645 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013656 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013659 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013661 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013664 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013667 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013669 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013672 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013675 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013678 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013681 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013684 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013687 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013690 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013692 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013695 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:29:58.014157 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013697 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013700 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013702 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013705 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013707 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013710 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013712 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013714 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013717 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013720 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013722 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013725 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013727 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013731 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013735 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013739 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013743 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013746 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013748 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:29:58.014666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013751 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013753 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013755 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013758 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013762 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013764 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013767 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013770 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013773 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013775 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013778 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013781 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013784 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013787 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013789 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013792 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013795 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013797 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013800 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:29:58.015159 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013804 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013806 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013808 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013811 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013814 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013817 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013819 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013821 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013824 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013827 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013829 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013832 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013834 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013837 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013839 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013842 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013844 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013847 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013850 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:29:58.015649 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013852 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013856 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013858 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013861 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013863 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013866 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.013869 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014682 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014689 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014692 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014696 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014698 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014701 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014704 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014708 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014719 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014722 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014725 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014727 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:29:58.016203 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014730 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014732 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014735 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014737 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014740 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014742 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014745 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014747 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014750 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014753 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014755 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014758 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014761 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014764 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014766 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014769 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014772 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014775 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014777 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014780 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:29:58.016698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014782 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014785 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014787 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014789 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014792 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014794 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014797 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014799 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014803 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014805 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014809 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014813 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014816 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014819 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014822 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014824 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014827 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014832 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014836 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:29:58.017194 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014838 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014841 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014844 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014847 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014849 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014852 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014854 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014857 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014860 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014862 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014865 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014867 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014870 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014872 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014875 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014877 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014880 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014882 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014884 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:29:58.017680 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014887 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014889 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014891 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014894 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014897 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014900 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014902 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014905 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014907 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014909 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014912 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014915 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014917 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014920 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014922 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014928 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014930 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014933 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014935 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:29:58.018145 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014938 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014940 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014942 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014945 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014947 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014949 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014952 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014954 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014957 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014959 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014962 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014964 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014967 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014970 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014973 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014975 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.014977 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015520 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015526 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015530 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:29:58.018595 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015533 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015536 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015539 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015542 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015545 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015547 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015550 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015552 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015556 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015558 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015561 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015563 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015568 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015571 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015573 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015575 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015578 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015580 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015583 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015585 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:29:58.019104 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015587 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015590 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015592 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015595 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015597 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015599 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015602 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015604 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015607 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015610 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015612 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015615 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015617 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015619 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015622 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015624 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015627 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015630 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015634 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:29:58.019573 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015637 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015639 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015642 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015645 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015647 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015651 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015653 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015656 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015659 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015661 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015664 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015667 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015671 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015675 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015678 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015680 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015682 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015685 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015687 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:29:58.020027 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015690 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015703 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015707 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015710 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015713 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015716 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015719 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015721 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015724 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015726 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015729 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015732 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015735 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015737 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015740 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015742 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015744 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015747 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015749 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:29:58.020475 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015752 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015755 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015757 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015760 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015762 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015764 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015767 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015769 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015772 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015774 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015777 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015779 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015781 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015784 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015786 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015789 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015791 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015793 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015796 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:29:58.020923 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015800 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015802 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015805 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015807 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015810 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015813 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.015815 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016802 2581 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016814 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016821 2581 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016826 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016830 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016834 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016838 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016842 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016845 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016848 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016852 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016856 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016859 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016862 2581 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016864 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016867 2581 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:29:58.021381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016870 2581 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016873 2581 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016877 2581 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016879 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016882 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016888 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016891 2581 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016895 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016898 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016903 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016906 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016909 2581 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016912 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016915 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016918 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016921 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016925 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016928 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016930 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016933 2581 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016936 2581 flags.go:64] FLAG: --enable-server="true" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016940 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016944 2581 flags.go:64] FLAG: --event-burst="100" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016948 2581 flags.go:64] FLAG: --event-qps="50" Oct 30 05:29:58.021913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016951 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016954 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016973 2581 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016979 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016982 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016986 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016989 2581 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016992 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016995 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.016998 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017001 2581 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017004 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017007 2581 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017011 2581 flags.go:64] FLAG: --feature-gates="" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017026 2581 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017029 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017032 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017035 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017039 2581 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017042 2581 flags.go:64] FLAG: --help="false" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017046 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017049 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017052 2581 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:29:58.022512 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017054 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017058 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017061 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017064 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017067 2581 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017070 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017073 2581 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017076 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017079 2581 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017081 2581 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017084 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017087 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017090 2581 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017093 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017096 2581 flags.go:64] FLAG: --lock-file="" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017098 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017101 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017104 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017110 2581 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017113 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017115 2581 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017118 2581 flags.go:64] FLAG: --logging-format="text" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017123 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017126 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:29:58.023122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017129 2581 flags.go:64] FLAG: --manifest-url="" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017132 2581 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017137 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017140 2581 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017144 2581 flags.go:64] FLAG: --max-pods="110" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017147 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017149 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017152 2581 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017155 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017158 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017161 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017163 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017171 2581 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017174 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017178 2581 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017181 2581 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017184 2581 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017187 2581 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017190 2581 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017193 2581 flags.go:64] FLAG: --port="10250" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017196 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017199 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08573b397eedecbd1" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017203 2581 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017205 2581 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017208 2581 flags.go:64] FLAG: --register-node="true" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017211 2581 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:29:58.023734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017215 2581 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017218 2581 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017221 2581 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017224 2581 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017229 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017232 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017235 2581 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017238 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017241 2581 flags.go:64] FLAG: --runonce="false" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017243 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017246 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017250 2581 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017253 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017255 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017258 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017261 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017264 2581 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017267 2581 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017270 2581 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017273 2581 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017277 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017280 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017283 2581 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017286 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017291 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:29:58.024376 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017294 2581 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017296 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017300 2581 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017303 2581 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017306 2581 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017309 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017312 2581 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017315 2581 flags.go:64] FLAG: --v="2" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017319 2581 flags.go:64] FLAG: --version="false" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017323 2581 flags.go:64] FLAG: --vmodule="" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017328 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017333 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017473 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017477 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017480 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017483 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017486 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017489 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017491 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017494 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017497 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017500 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:29:58.024967 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017504 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017508 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017511 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017514 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017517 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017522 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017524 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017527 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017529 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017532 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017534 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017537 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017540 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017542 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017545 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017547 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017549 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017552 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017554 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:29:58.025506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017557 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017559 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017563 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017565 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017568 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017571 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017573 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017576 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017578 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017581 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017583 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017586 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017589 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017591 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017594 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017596 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017599 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017601 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017605 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017607 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:29:58.025989 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017610 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017612 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017615 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017617 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017620 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017622 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017625 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017627 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017630 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017632 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017635 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017637 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017640 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017642 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017646 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017649 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017651 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017654 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017656 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017659 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:29:58.026508 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017661 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017663 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017666 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017668 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017672 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017674 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017677 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017679 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017682 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017685 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017688 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017691 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017694 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017696 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017700 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017703 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017707 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017709 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017712 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017714 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:29:58.027012 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017717 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017720 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017722 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017725 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017727 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017730 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017733 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017736 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017738 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017741 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017743 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017746 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017749 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017752 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017754 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017757 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:29:58.027520 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017759 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.017785 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017828 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017833 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017836 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017839 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017843 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017845 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017848 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017850 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017853 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017856 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017858 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017860 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017864 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:29:58.027905 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017868 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017872 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017875 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017878 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017881 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017883 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017886 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017889 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017893 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017895 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017898 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017900 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017903 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017905 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017908 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017911 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017913 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017915 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017918 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:29:58.028290 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017921 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017924 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017926 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017929 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017932 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017935 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017938 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017940 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017942 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017945 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017947 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017950 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017952 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017955 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017957 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017959 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017962 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017965 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017967 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017969 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:29:58.028735 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017972 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017976 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017979 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017981 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017984 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017986 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017989 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017991 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017994 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017996 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.017999 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018001 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018004 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018007 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018009 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018012 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018027 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018030 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018033 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:29:58.029220 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018035 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018038 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018040 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018043 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018046 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018048 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018051 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018053 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018056 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018058 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018061 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018064 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018067 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018069 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018075 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018079 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018082 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018084 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018088 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018091 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:29:58.029666 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018094 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018096 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018099 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018101 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018104 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018106 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018109 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018112 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018114 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018118 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018121 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018123 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018126 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018129 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.018132 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.025277 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.025291 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.027208 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.027217 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028003 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:29:58.030169 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028223 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028226 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028229 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028384 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028467 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028471 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028473 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028476 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028479 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028481 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028567 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028722 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028726 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028728 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028731 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028734 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028737 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028974 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.028978 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029184 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:29:58.030647 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029191 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029193 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029196 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029199 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029203 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029206 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029208 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029211 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029214 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029216 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029219 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029223 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029227 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029230 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029235 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029237 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029240 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029242 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029247 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:29:58.031133 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029249 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029252 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029254 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029257 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029261 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029263 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029266 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029268 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029271 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029275 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029278 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029281 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029283 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029299 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029303 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029305 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029310 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029313 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029315 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:29:58.031576 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029318 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029320 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029326 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029330 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029334 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029339 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029342 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029345 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029347 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029352 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029355 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029357 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029359 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029364 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029367 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029370 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029372 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029377 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029379 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:29:58.032069 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029382 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029384 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029387 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029391 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029394 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029396 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029399 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029402 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029412 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029415 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029420 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029422 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029425 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029427 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029432 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029435 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029438 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029457 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029462 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029465 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:29:58.032517 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029468 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029470 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029475 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029478 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029480 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029483 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029487 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029490 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.029523 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029739 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029744 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029747 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029751 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029757 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029761 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029764 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:29:58.032977 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029769 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029771 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029774 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029776 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029779 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029789 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029791 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029796 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029799 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029801 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029804 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029807 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029813 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029815 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029818 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029820 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029825 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029828 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029832 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:29:58.033363 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029835 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029840 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029842 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029845 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029847 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029851 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029854 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029857 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029859 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029861 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029866 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029868 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029871 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029873 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029876 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029880 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029883 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029885 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029888 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029898 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:29:58.033818 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029901 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029906 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029908 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029911 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029914 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029919 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029922 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029924 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029927 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029929 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029933 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029936 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029938 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029941 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029945 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029947 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029950 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029952 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029955 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029957 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:29:58.034420 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029960 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029962 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029965 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029967 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029970 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029974 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029976 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029979 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029981 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029983 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029988 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.029990 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030007 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030009 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030012 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030030 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030033 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030036 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030040 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030043 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:29:58.034942 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030046 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030050 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030053 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030055 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030058 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030062 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030065 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030068 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030070 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030075 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030077 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030079 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030084 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030086 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030089 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030091 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030096 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030098 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030101 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:29:58.035429 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030103 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.030134 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030193 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030198 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030201 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030206 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030216 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030219 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030221 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030226 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030229 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030232 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030234 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030239 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030241 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030244 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:29:58.036008 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030248 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030251 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030254 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030256 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030261 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030263 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030266 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030271 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030273 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030276 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030278 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030282 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030285 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030287 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030290 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030294 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030296 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030299 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030301 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030306 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:29:58.036441 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030308 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030312 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030317 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030328 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030331 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030334 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030339 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030341 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030343 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030346 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030350 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030353 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030356 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030360 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030362 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030365 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030367 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030370 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030372 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:29:58.037007 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030375 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030379 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030382 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030385 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030391 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030394 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030397 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030401 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030404 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030407 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030409 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030414 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030416 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030418 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030421 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030426 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030428 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030439 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030442 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030446 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:29:58.037506 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030449 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030451 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030453 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030458 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030462 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030464 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030469 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030471 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030474 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030476 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030480 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030483 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030485 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030490 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030492 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030495 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030497 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030501 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030503 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:29:58.038282 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030506 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030508 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030513 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030515 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030518 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030520 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030524 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030527 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030529 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030534 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030538 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030551 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030555 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:58.030558 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.030622 2581 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.034391 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.035410 2581 server.go:1014] "Starting client certificate rotation" Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.035520 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:29:58.039037 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.035565 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:29:58.041476 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.041453 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:29:58.042708 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.042675 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:29:58.050829 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.050811 2581 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:29:58.054364 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.054350 2581 log.go:25] "Validated CRI v1 image API" Oct 30 05:29:58.055193 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.055168 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:29:58.060071 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.060052 2581 fs.go:135] Filesystem UUIDs: map[1b181e71-cc84-4571-a13c-b4605bc09f04:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8f84bfb4-0330-42ed-abbd-b96c7bfae532:/dev/nvme0n1p3] Oct 30 05:29:58.060122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.060071 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 05:29:58.064331 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.064218 2581 manager.go:217] Machine: {Timestamp:2026-10-30 05:29:58.063730651 +0000 UTC m=+0.384723818 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100721 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a0b59bdbaf143c2eeda045376a42 SystemUUID:ec28a0b5-9bdb-af14-3c2e-eda045376a42 BootID:dcc054e7-04fe-4fc1-829b-13d4bd128279 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e3:97:4d:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e3:97:4d:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:2c:02:3f:ef:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:29:58.064331 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.064321 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:29:58.064459 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.064443 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:29:58.067495 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.067468 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:29:58.067626 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.067497 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:29:58.067673 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.067629 2581 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:29:58.067673 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.067637 2581 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:29:58.067673 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.067647 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:29:58.067673 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.067657 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:29:58.068652 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.068642 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:29:58.073282 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.073270 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:29:58.073400 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.073391 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:29:58.074453 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.074443 2581 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:29:58.074508 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.074458 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:29:58.074508 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.074473 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:29:58.074508 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.074481 2581 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:29:58.074508 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.074498 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:29:58.075042 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.075011 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:29:58.077478 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.077451 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:29:58.085142 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.085124 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:29:58.086083 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.086068 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:29:58.086149 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.086109 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:29:58.086296 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.086284 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:29:58.086354 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.086303 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:29:58.086354 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.086313 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:29:58.086354 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.086322 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:29:58.086354 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.086331 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:29:58.086354 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.086340 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:29:58.087011 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.087001 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:29:58.087078 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.087031 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:29:58.087078 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.087043 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:29:58.087078 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.087053 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:29:58.087078 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.087066 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:29:58.087307 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.087081 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:29:58.087994 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.087983 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:29:58.088061 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.087998 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:29:58.091874 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.091858 2581 server.go:1263] "Started kubelet" Oct 30 05:29:58.091965 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.091919 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:29:58.092063 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.092004 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:29:58.092106 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.092082 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:29:58.093476 ip-10-0-138-200 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:29:58.094073 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.093476 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:29:58.094442 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.094389 2581 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:29:58.100193 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.100173 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 05:29:58.102453 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.102325 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:29:58.103423 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.103406 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:29:58.103517 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.103502 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:58.104102 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104087 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:29:58.104102 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104088 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:29:58.104245 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104106 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:29:58.104245 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104181 2581 factory.go:55] Registering systemd factory Oct 30 05:29:58.104245 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104197 2581 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:29:58.104245 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104240 2581 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:29:58.104245 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104248 2581 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:29:58.104473 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104463 2581 factory.go:153] Registering CRI-O factory Oct 30 05:29:58.104520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104476 2581 factory.go:223] Registration of the crio container factory successfully Oct 30 05:29:58.104572 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104519 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:29:58.104572 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104537 2581 factory.go:103] Registering Raw factory Oct 30 05:29:58.104572 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.104554 2581 manager.go:1199] Started watching for new ooms in manager Oct 30 05:29:58.105398 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.105384 2581 manager.go:319] Starting recovery of all containers Oct 30 05:29:58.116787 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.116764 2581 manager.go:324] Recovery completed Oct 30 05:29:58.119525 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.119494 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 05:29:58.119596 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.119570 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-200.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 05:29:58.120826 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.119495 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-200.ec2.internal.18e33796f36714e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-200.ec2.internal,},FirstTimestamp:2026-10-30 05:29:58.091830502 +0000 UTC m=+0.412823667,LastTimestamp:2026-10-30 05:29:58.091830502 +0000 UTC m=+0.412823667,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-200.ec2.internal,}" Oct 30 05:29:58.121797 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.121784 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:29:58.123502 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.123484 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:29:58.123560 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.123512 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:29:58.123560 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.123523 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:29:58.124045 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.124030 2581 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:29:58.124045 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.124042 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:29:58.124045 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.124045 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:29:58.124222 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.124060 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:29:58.124681 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.124667 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:29:58.124725 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.124693 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:29:58.124725 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.124703 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:29:58.126220 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.126207 2581 policy_none.go:50] "Start" Oct 30 05:29:58.126294 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.126226 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:29:58.126294 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.126239 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:29:58.128194 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.128181 2581 policy_none.go:44] "Start" Oct 30 05:29:58.130530 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.130412 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:29:58.138171 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.138094 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-200.ec2.internal.18e33796f54a52c9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-200.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-200.ec2.internal,},FirstTimestamp:2026-10-30 05:29:58.123500233 +0000 UTC m=+0.444493398,LastTimestamp:2026-10-30 05:29:58.123500233 +0000 UTC m=+0.444493398,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-200.ec2.internal,}" Oct 30 05:29:58.138260 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.138201 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 05:29:58.138487 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.138464 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 05:29:58.138564 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.138530 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 05:29:58.159629 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.159564 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-200.ec2.internal.18e33796f54a9543 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-200.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-200.ec2.internal,},FirstTimestamp:2026-10-30 05:29:58.123517251 +0000 UTC m=+0.444510416,LastTimestamp:2026-10-30 05:29:58.123517251 +0000 UTC m=+0.444510416,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-200.ec2.internal,}" Oct 30 05:29:58.163155 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.163134 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mf5ql" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.168601 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.171078 2581 manager.go:341] "Starting Device Plugin manager" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.171112 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.171122 2581 server.go:87] "Starting device plugin registration server" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.171350 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.171362 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.171461 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.171542 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.171551 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.172037 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.172098 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.173158 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-200.ec2.internal.18e33796f54ab7ee default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-200.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-200.ec2.internal,},FirstTimestamp:2026-10-30 05:29:58.123526126 +0000 UTC m=+0.444519290,LastTimestamp:2026-10-30 05:29:58.123526126 +0000 UTC m=+0.444519290,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-200.ec2.internal,}" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.175190 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mf5ql" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.177928 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.179235 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.179253 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.179271 2581 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:29:58.184520 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.179299 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 05:29:58.216903 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.216887 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:29:58.272475 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.272403 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:29:58.273325 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.273306 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:29:58.273427 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.273340 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:29:58.273427 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.273354 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:29:58.273427 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.273391 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.279799 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.279777 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"] Oct 30 05:29:58.279877 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.279848 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:29:58.280638 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.280624 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:29:58.280706 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.280650 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:29:58.280706 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.280660 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:29:58.282858 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.282847 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:29:58.283009 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.282994 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.283093 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.283052 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:29:58.283608 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.283587 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:29:58.283687 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.283611 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:29:58.283687 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.283639 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:29:58.283687 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.283650 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:29:58.283687 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.283619 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:29:58.283815 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.283711 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:29:58.285715 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.285699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.285787 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.285724 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:29:58.286437 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.286422 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:29:58.286515 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.286451 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:29:58.286515 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.286465 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:29:58.296118 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.296105 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.296201 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.296124 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-200.ec2.internal\": node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:58.305469 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.305445 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.305469 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.305470 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.305595 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.305485 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.315734 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.315717 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-200.ec2.internal\" not found" node="ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.318258 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.318239 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-200.ec2.internal\" not found" node="ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.345519 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.345496 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:58.406203 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.406177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.406203 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.406204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.406379 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.406221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.406379 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.406246 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.406379 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.406276 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.406379 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.406283 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.445600 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.445565 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:58.545681 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.545623 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:58.621080 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.621051 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.623199 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:58.623181 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:29:58.645753 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.645726 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:58.745980 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.745945 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:58.846473 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.846398 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:58.946890 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:58.946858 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:59.035408 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.035372 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 05:29:59.035839 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.035509 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 05:29:59.047622 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:59.047599 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:59.147715 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:29:59.147678 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:29:59.181064 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.181005 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:24:59 +0000 UTC" deadline="2028-04-16 00:42:34.901914292 +0000 UTC" Oct 30 05:29:59.181064 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.181056 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12811h12m35.720861729s" Oct 30 05:29:59.209495 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.209471 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:29:59.210628 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.210612 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:29:59.210696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.210649 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:29:59.210696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.210661 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:29:59.242904 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.242873 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:29:59.275730 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:59.275690 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d685204da0cc5fa69ed201b4cfe0a4.slice/crio-dfc163cc70ad703354b5534c2a01765a1b84d0b36b3faa95673a4e97ab17619e WatchSource:0}: Error finding container dfc163cc70ad703354b5534c2a01765a1b84d0b36b3faa95673a4e97ab17619e: Status 404 returned error can't find the container with id dfc163cc70ad703354b5534c2a01765a1b84d0b36b3faa95673a4e97ab17619e Oct 30 05:29:59.276121 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:29:59.276099 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd03e27bf6704eb151f6eb79c368f447.slice/crio-6e680adc9c3cba9823577b254fd10ef66223c00ece6a78ce082dcb9d97b39d93 WatchSource:0}: Error finding container 6e680adc9c3cba9823577b254fd10ef66223c00ece6a78ce082dcb9d97b39d93: Status 404 returned error can't find the container with id 6e680adc9c3cba9823577b254fd10ef66223c00ece6a78ce082dcb9d97b39d93 Oct 30 05:29:59.281419 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.281406 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:29:59.291058 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.291041 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:29:59.303801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.303787 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:29:59.325619 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.325602 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:29:59.326752 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.326739 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:29:59.351164 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:29:59.351144 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:30:00.678863 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:00.678837 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:01.104765 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.104690 2581 apiserver.go:52] "Watching apiserver" Oct 30 05:30:01.132533 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.132499 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 05:30:01.145529 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.145506 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:30:01.145949 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.145926 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-gzrqp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal","openshift-multus/multus-additional-cni-plugins-kt95j","openshift-multus/network-metrics-daemon-mtw44","openshift-network-diagnostics/network-check-target-bmv92","openshift-network-operator/iptables-alerter-78m84","openshift-ovn-kubernetes/ovnkube-node-d2qsz","kube-system/global-pull-secret-syncer-g6mhk","openshift-dns/node-resolver-trbn7","openshift-image-registry/node-ca-vltvd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8","openshift-multus/multus-jlq5d","kube-system/konnectivity-agent-scqnq","kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"] Oct 30 05:30:01.149073 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.149056 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.151074 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.151048 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:01.153618 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.153588 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.155900 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.155876 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:01.156030 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.155981 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:01.158120 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.158094 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:01.158221 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.158176 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:01.160289 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.160263 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.163189 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.163164 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.164310 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.164289 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:30:01.164519 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.164443 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:30:01.164519 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.164444 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:30:01.164519 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.164518 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:30:01.164713 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.164527 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:30:01.164713 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.164595 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5cj6p\"" Oct 30 05:30:01.164713 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.164671 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:30:01.165357 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.165336 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.166367 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.165884 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:30:01.166367 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.166003 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:30:01.166367 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.166234 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c6sjw\"" Oct 30 05:30:01.168109 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.168087 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:30:01.168204 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.168192 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:30:01.170198 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.169642 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.170198 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.169711 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:01.172259 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.172236 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:30:01.172388 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.172372 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:30:01.172587 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.172564 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:30:01.175634 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.175618 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.176590 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.176567 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wnglw\"" Oct 30 05:30:01.176699 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.176682 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.178789 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.178772 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.181475 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.181433 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:30:01.181681 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.181656 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4f4km" Oct 30 05:30:01.191237 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.191155 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4f4km" Oct 30 05:30:01.215115 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.215068 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" event={"ID":"fd03e27bf6704eb151f6eb79c368f447","Type":"ContainerStarted","Data":"6e680adc9c3cba9823577b254fd10ef66223c00ece6a78ce082dcb9d97b39d93"} Oct 30 05:30:01.216571 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.216546 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerStarted","Data":"dfc163cc70ad703354b5534c2a01765a1b84d0b36b3faa95673a4e97ab17619e"} Oct 30 05:30:01.234760 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.234737 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:30:01.246053 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246013 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-cnibin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.246165 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246055 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-multus-certs\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.246165 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246083 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48wsv\" (UniqueName: \"kubernetes.io/projected/510573e7-a380-415f-9609-18470c856703-kube-api-access-48wsv\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.246165 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246102 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-config\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.246165 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-tmp\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.246165 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-etc-kubernetes\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.246165 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246160 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-netns\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.246413 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246174 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-host\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.246413 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.246413 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:01.246413 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246265 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-var-lib-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.246413 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-log-socket\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.246413 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246302 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74f707f9-cd75-4d5f-bab3-3ead92d87904-konnectivity-ca\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:01.246413 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246330 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-cnibin\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.246413 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246383 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246414 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3d3980e-8842-4c87-a275-8bf0da706fba-hosts-file\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246476 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-netns\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246492 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6f78\" (UniqueName: \"kubernetes.io/projected/e0a3b194-c453-404f-92a4-0b1ab87f7699-kube-api-access-v6f78\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hcv9\" (UniqueName: \"kubernetes.io/projected/671efddc-7c80-4be5-827d-0c227985c013-kube-api-access-8hcv9\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246524 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-modprobe-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246538 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-cni-binary-copy\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3d3980e-8842-4c87-a275-8bf0da706fba-tmp-dir\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246571 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-socket-dir-parent\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246599 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-hostroot\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246678 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-k8s-cni-cncf-io\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovn-node-metrics-cert\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.246712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246715 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-run\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-sys\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-lib-modules\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246765 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-var-lib-kubelet\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-serviceca\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-registration-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-host-slash\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-kubelet\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246871 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-systemd-units\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-systemd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246926 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-device-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246952 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-bin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.246976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-system-cni-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247029 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysconfig\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.247124 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247095 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-kubernetes\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247107 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-etc-selinux\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74f707f9-cd75-4d5f-bab3-3ead92d87904-agent-certs\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247167 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-etc-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247185 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-ovn\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-iptables-alerter-script\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247243 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-kubelet\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247259 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7q66\" (UniqueName: \"kubernetes.io/projected/f3d3980e-8842-4c87-a275-8bf0da706fba-kube-api-access-l7q66\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247282 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-kubelet-config\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-dbus\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-bin\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttx6t\" (UniqueName: \"kubernetes.io/projected/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-kube-api-access-ttx6t\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247388 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-conf\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-host\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdk9\" (UniqueName: \"kubernetes.io/projected/62348232-2a4d-4ed2-8830-10f547a59fea-kube-api-access-bjdk9\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.247677 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247430 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-daemon-config\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247443 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247479 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78ks\" (UniqueName: \"kubernetes.io/projected/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-kube-api-access-n78ks\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247499 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-netd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-env-overrides\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247556 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.247598 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-etc-tuned\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248028 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-slash\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248107 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248231 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.248301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-systemd\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.248848 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248312 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-conf-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.248848 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248341 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-sys-fs\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.248848 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248383 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v974p\" (UniqueName: \"kubernetes.io/projected/12fe3fee-9496-4b7b-8923-d19c585cc4be-kube-api-access-v974p\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.248848 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-script-lib\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.248848 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgk6\" (UniqueName: \"kubernetes.io/projected/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-kube-api-access-spgk6\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.248848 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-multus\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.248848 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-node-log\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.248848 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-socket-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.248848 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.248622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-system-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.264747 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.264717 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:30:01.264875 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.264802 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:01.265001 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.264981 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:01.265211 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265195 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:30:01.265326 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265302 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:01.265442 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265357 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:30:01.265525 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265442 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:30:01.265525 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265491 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w87jr\"" Oct 30 05:30:01.265609 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265541 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:30:01.265656 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265626 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-76x48\"" Oct 30 05:30:01.265656 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265635 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nnkr\"" Oct 30 05:30:01.265739 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265662 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:01.265739 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265668 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:30:01.265739 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265688 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:30:01.265739 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265194 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:30:01.265896 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265762 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:30:01.265896 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265783 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt4k5\"" Oct 30 05:30:01.265896 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265796 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:30:01.265896 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.265868 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-twkbr\"" Oct 30 05:30:01.266062 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.266017 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-88sr6\"" Oct 30 05:30:01.349175 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.349147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48wsv\" (UniqueName: \"kubernetes.io/projected/510573e7-a380-415f-9609-18470c856703-kube-api-access-48wsv\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.349175 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.349180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-config\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.349423 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.349210 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-tmp\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.349423 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.349233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-etc-kubernetes\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.349423 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.349258 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-netns\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.349423 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.349272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-host\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350024 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-netns\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350125 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-etc-kubernetes\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-var-lib-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350305 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-log-socket\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74f707f9-cd75-4d5f-bab3-3ead92d87904-konnectivity-ca\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350377 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350398 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-cnibin\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3d3980e-8842-4c87-a275-8bf0da706fba-hosts-file\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350575 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-host\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350649 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-netns\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6f78\" (UniqueName: \"kubernetes.io/projected/e0a3b194-c453-404f-92a4-0b1ab87f7699-kube-api-access-v6f78\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350698 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-cnibin\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.350913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350708 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hcv9\" (UniqueName: \"kubernetes.io/projected/671efddc-7c80-4be5-827d-0c227985c013-kube-api-access-8hcv9\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350918 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.350987 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3d3980e-8842-4c87-a275-8bf0da706fba-hosts-file\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351092 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-log-socket\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351107 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351159 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-var-lib-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351163 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-config\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.351266 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-modprobe-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.351344 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs podName:671efddc-7c80-4be5-827d-0c227985c013 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:01.85132499 +0000 UTC m=+3.142878010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs") pod "network-metrics-daemon-mtw44" (UID: "671efddc-7c80-4be5-827d-0c227985c013") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351381 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-cni-binary-copy\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3d3980e-8842-4c87-a275-8bf0da706fba-tmp-dir\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-socket-dir-parent\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-hostroot\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351516 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351597 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74f707f9-cd75-4d5f-bab3-3ead92d87904-konnectivity-ca\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351644 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-netns\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.351801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351755 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-socket-dir-parent\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.351998 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3d3980e-8842-4c87-a275-8bf0da706fba-tmp-dir\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-k8s-cni-cncf-io\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352111 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352144 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovn-node-metrics-cert\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352141 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-modprobe-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352210 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-hostroot\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352206 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-cni-binary-copy\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-run\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352247 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-k8s-cni-cncf-io\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-sys\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352265 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-run\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-lib-modules\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-var-lib-kubelet\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352321 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-sys\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-serviceca\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-registration-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352400 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-host-slash\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.352826 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352422 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-lib-modules\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352453 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-host-slash\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352506 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-registration-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352513 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-kubelet\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-systemd-units\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352561 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-var-lib-kubelet\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-systemd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352605 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-device-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352620 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-kubelet\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352632 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-systemd-units\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-bin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-system-cni-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352688 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-device-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352672 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-systemd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.352706 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353112 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353164 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-system-cni-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.353611 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353224 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysconfig\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-kubernetes\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353381 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysconfig\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353398 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-serviceca\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-etc-selinux\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74f707f9-cd75-4d5f-bab3-3ead92d87904-agent-certs\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-etc-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-ovn\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-iptables-alerter-script\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353622 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-bin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353648 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-kubelet\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7q66\" (UniqueName: \"kubernetes.io/projected/f3d3980e-8842-4c87-a275-8bf0da706fba-kube-api-access-l7q66\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-kubelet-config\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-dbus\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353740 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-etc-selinux\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.354333 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353781 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-bin\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353788 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-kubernetes\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353813 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttx6t\" (UniqueName: \"kubernetes.io/projected/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-kube-api-access-ttx6t\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-conf\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-host\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdk9\" (UniqueName: \"kubernetes.io/projected/62348232-2a4d-4ed2-8830-10f547a59fea-kube-api-access-bjdk9\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.353964 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-daemon-config\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78ks\" (UniqueName: \"kubernetes.io/projected/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-kube-api-access-n78ks\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-netd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-env-overrides\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354120 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-etc-tuned\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-slash\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354240 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355161 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-systemd\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-conf-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-sys-fs\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v974p\" (UniqueName: \"kubernetes.io/projected/12fe3fee-9496-4b7b-8923-d19c585cc4be-kube-api-access-v974p\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354448 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-kubelet-config\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-script-lib\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgk6\" (UniqueName: \"kubernetes.io/projected/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-kube-api-access-spgk6\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354634 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-host\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-multus\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354696 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-multus\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-node-log\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354858 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-ovn\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354906 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-etc-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354955 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-netd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.354967 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-socket-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.355294 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-env-overrides\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.355356 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.355823 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.355393 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-socket-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.355542 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-iptables-alerter-script\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.355549 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-script-lib\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.355621 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-node-log\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.355777 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-kubelet\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.355893 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356009 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356017 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-bin\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356019 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-dbus\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356042 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356104 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-slash\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356108 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-conf\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.356278 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-system-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356343 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-system-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.356400 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret podName:56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde nodeName:}" failed. No retries permitted until 2026-10-30 05:30:01.856364164 +0000 UTC m=+3.147917180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret") pod "global-pull-secret-syncer-g6mhk" (UID: "56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-cnibin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.356696 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356489 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-systemd\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.357445 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356502 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-daemon-config\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.357445 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356517 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-multus-certs\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.357445 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356513 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-conf-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.357445 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356422 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.357445 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356639 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.357445 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356647 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-sys-fs\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.357445 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356725 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-cnibin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.357445 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.356926 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-multus-certs\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.358524 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.358501 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-tmp\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.358622 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.358607 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovn-node-metrics-cert\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.364013 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.360006 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74f707f9-cd75-4d5f-bab3-3ead92d87904-agent-certs\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:01.364013 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.363976 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-etc-tuned\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.372425 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.372404 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:01.372528 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.372430 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:01.372528 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.372441 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5c926 for pod openshift-network-diagnostics/network-check-target-bmv92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:01.372528 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.372519 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926 podName:0d304ba5-b889-412c-ae7d-b26f04222714 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:01.87250468 +0000 UTC m=+3.164057705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5c926" (UniqueName: "kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926") pod "network-check-target-bmv92" (UID: "0d304ba5-b889-412c-ae7d-b26f04222714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:01.375820 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.375509 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48wsv\" (UniqueName: \"kubernetes.io/projected/510573e7-a380-415f-9609-18470c856703-kube-api-access-48wsv\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.375820 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.375700 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hcv9\" (UniqueName: \"kubernetes.io/projected/671efddc-7c80-4be5-827d-0c227985c013-kube-api-access-8hcv9\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:01.376401 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.376344 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6f78\" (UniqueName: \"kubernetes.io/projected/e0a3b194-c453-404f-92a4-0b1ab87f7699-kube-api-access-v6f78\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.377913 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.377889 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v974p\" (UniqueName: \"kubernetes.io/projected/12fe3fee-9496-4b7b-8923-d19c585cc4be-kube-api-access-v974p\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.379005 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.378986 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgk6\" (UniqueName: \"kubernetes.io/projected/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-kube-api-access-spgk6\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.379088 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.379061 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttx6t\" (UniqueName: \"kubernetes.io/projected/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-kube-api-access-ttx6t\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.379145 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.379064 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdk9\" (UniqueName: \"kubernetes.io/projected/62348232-2a4d-4ed2-8830-10f547a59fea-kube-api-access-bjdk9\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.380332 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.380314 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78ks\" (UniqueName: \"kubernetes.io/projected/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-kube-api-access-n78ks\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.383658 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.383635 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7q66\" (UniqueName: \"kubernetes.io/projected/f3d3980e-8842-4c87-a275-8bf0da706fba-kube-api-access-l7q66\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.475284 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.475240 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:01.479942 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.479913 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:01.489739 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:01.489708 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f707f9_cd75_4d5f_bab3_3ead92d87904.slice/crio-e8a892ff18a4642b84b08a0b8919611115d7933c8305a9419ebd8fe783b0b4a1 WatchSource:0}: Error finding container e8a892ff18a4642b84b08a0b8919611115d7933c8305a9419ebd8fe783b0b4a1: Status 404 returned error can't find the container with id e8a892ff18a4642b84b08a0b8919611115d7933c8305a9419ebd8fe783b0b4a1 Oct 30 05:30:01.490748 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:01.490728 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc9f3ff_b5e8_4031_8a21_301c46e1bcf3.slice/crio-fb7d7b30f6715d9058a8efe8e704cc3bbe1b31e21df705683c085d5eae048859 WatchSource:0}: Error finding container fb7d7b30f6715d9058a8efe8e704cc3bbe1b31e21df705683c085d5eae048859: Status 404 returned error can't find the container with id fb7d7b30f6715d9058a8efe8e704cc3bbe1b31e21df705683c085d5eae048859 Oct 30 05:30:01.496650 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.496491 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:01.498890 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.498841 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:01.501480 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.501441 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:01.503687 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.503637 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:01.507760 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:01.507730 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db8ada4_0aec_4b8d_844b_e7d9abbe0850.slice/crio-407cf142ad3eab13473b854d83ff337fd6ad9d8dfb998eb36353b938b41c87b6 WatchSource:0}: Error finding container 407cf142ad3eab13473b854d83ff337fd6ad9d8dfb998eb36353b938b41c87b6: Status 404 returned error can't find the container with id 407cf142ad3eab13473b854d83ff337fd6ad9d8dfb998eb36353b938b41c87b6 Oct 30 05:30:01.511372 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.511352 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:01.511668 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:01.511582 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510573e7_a380_415f_9609_18470c856703.slice/crio-f09805b6662dd398b6fc65a665545903bcf7ff82716dfec8c9e494cf73c83a4a WatchSource:0}: Error finding container f09805b6662dd398b6fc65a665545903bcf7ff82716dfec8c9e494cf73c83a4a: Status 404 returned error can't find the container with id f09805b6662dd398b6fc65a665545903bcf7ff82716dfec8c9e494cf73c83a4a Oct 30 05:30:01.517732 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:01.517704 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62348232_2a4d_4ed2_8830_10f547a59fea.slice/crio-ea4c74fbffd46aa050b088a2974c75354bbdd71d5a411d2884d647598b869cac WatchSource:0}: Error finding container ea4c74fbffd46aa050b088a2974c75354bbdd71d5a411d2884d647598b869cac: Status 404 returned error can't find the container with id ea4c74fbffd46aa050b088a2974c75354bbdd71d5a411d2884d647598b869cac Oct 30 05:30:01.522223 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:01.522189 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d3980e_8842_4c87_a275_8bf0da706fba.slice/crio-00b270c441897dd0bfe434e21a437881868290a414a56e97e433147e18c2eb5e WatchSource:0}: Error finding container 00b270c441897dd0bfe434e21a437881868290a414a56e97e433147e18c2eb5e: Status 404 returned error can't find the container with id 00b270c441897dd0bfe434e21a437881868290a414a56e97e433147e18c2eb5e Oct 30 05:30:01.524183 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.524165 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:01.525861 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:01.525831 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7c2ae2_e9e9_40c2_a05f_f4c05d3fcc37.slice/crio-f34759dd8e5f7cb9a690cf8c0e822e53cf60f041a8291a34b300223e4f21afc3 WatchSource:0}: Error finding container f34759dd8e5f7cb9a690cf8c0e822e53cf60f041a8291a34b300223e4f21afc3: Status 404 returned error can't find the container with id f34759dd8e5f7cb9a690cf8c0e822e53cf60f041a8291a34b300223e4f21afc3 Oct 30 05:30:01.526648 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.526573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jlq5d" Oct 30 05:30:01.537835 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.537799 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:01.538505 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:01.538453 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fe3fee_9496_4b7b_8923_d19c585cc4be.slice/crio-a83625753bb547479b8cfc9edc087e83052e20c023f8f582568ce55fc0a4bfb9 WatchSource:0}: Error finding container a83625753bb547479b8cfc9edc087e83052e20c023f8f582568ce55fc0a4bfb9: Status 404 returned error can't find the container with id a83625753bb547479b8cfc9edc087e83052e20c023f8f582568ce55fc0a4bfb9 Oct 30 05:30:01.539253 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:01.539212 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a3b194_c453_404f_92a4_0b1ab87f7699.slice/crio-4844eaf59c98ad95cf2e0b685973a123bd1af956d0d731de12eeeaee3ea1fc11 WatchSource:0}: Error finding container 4844eaf59c98ad95cf2e0b685973a123bd1af956d0d731de12eeeaee3ea1fc11: Status 404 returned error can't find the container with id 4844eaf59c98ad95cf2e0b685973a123bd1af956d0d731de12eeeaee3ea1fc11 Oct 30 05:30:01.861856 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.861828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:01.861951 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.861932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:01.861991 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.861955 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:01.862347 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.862333 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs podName:671efddc-7c80-4be5-827d-0c227985c013 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:02.861993911 +0000 UTC m=+4.153546913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs") pod "network-metrics-daemon-mtw44" (UID: "671efddc-7c80-4be5-827d-0c227985c013") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:01.864760 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.864533 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:01.864858 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.864787 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret podName:56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde nodeName:}" failed. No retries permitted until 2026-10-30 05:30:02.864768426 +0000 UTC m=+4.156321444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret") pod "global-pull-secret-syncer-g6mhk" (UID: "56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:01.962726 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:01.962694 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:01.962873 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.962830 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:01.962873 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.962845 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:01.962873 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.962854 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5c926 for pod openshift-network-diagnostics/network-check-target-bmv92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:01.963015 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:01.962910 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926 podName:0d304ba5-b889-412c-ae7d-b26f04222714 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:02.962896926 +0000 UTC m=+4.254449927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c926" (UniqueName: "kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926") pod "network-check-target-bmv92" (UID: "0d304ba5-b889-412c-ae7d-b26f04222714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:02.194974 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.194622 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:25:01 +0000 UTC" deadline="2028-06-04 09:16:25.821796169 +0000 UTC" Oct 30 05:30:02.194974 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.194680 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13995h46m23.627119551s" Oct 30 05:30:02.231541 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.231266 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vltvd" event={"ID":"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37","Type":"ContainerStarted","Data":"f34759dd8e5f7cb9a690cf8c0e822e53cf60f041a8291a34b300223e4f21afc3"} Oct 30 05:30:02.240252 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.240222 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" event={"ID":"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3","Type":"ContainerStarted","Data":"fb7d7b30f6715d9058a8efe8e704cc3bbe1b31e21df705683c085d5eae048859"} Oct 30 05:30:02.242474 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.242436 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-scqnq" event={"ID":"74f707f9-cd75-4d5f-bab3-3ead92d87904","Type":"ContainerStarted","Data":"e8a892ff18a4642b84b08a0b8919611115d7933c8305a9419ebd8fe783b0b4a1"} Oct 30 05:30:02.248317 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.247374 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" event={"ID":"fd03e27bf6704eb151f6eb79c368f447","Type":"ContainerStarted","Data":"f212a1d11ee683823ab0c862bcd91e9cd98b5bb668da4dc1104aa7cae122840f"} Oct 30 05:30:02.252050 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.251970 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jlq5d" event={"ID":"e0a3b194-c453-404f-92a4-0b1ab87f7699","Type":"ContainerStarted","Data":"4844eaf59c98ad95cf2e0b685973a123bd1af956d0d731de12eeeaee3ea1fc11"} Oct 30 05:30:02.254534 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.254442 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-trbn7" event={"ID":"f3d3980e-8842-4c87-a275-8bf0da706fba","Type":"ContainerStarted","Data":"00b270c441897dd0bfe434e21a437881868290a414a56e97e433147e18c2eb5e"} Oct 30 05:30:02.257383 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.257323 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" event={"ID":"62348232-2a4d-4ed2-8830-10f547a59fea","Type":"ContainerStarted","Data":"ea4c74fbffd46aa050b088a2974c75354bbdd71d5a411d2884d647598b869cac"} Oct 30 05:30:02.260009 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.259933 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt95j" event={"ID":"510573e7-a380-415f-9609-18470c856703","Type":"ContainerStarted","Data":"f09805b6662dd398b6fc65a665545903bcf7ff82716dfec8c9e494cf73c83a4a"} Oct 30 05:30:02.265192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.265156 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-78m84" event={"ID":"9db8ada4-0aec-4b8d-844b-e7d9abbe0850","Type":"ContainerStarted","Data":"407cf142ad3eab13473b854d83ff337fd6ad9d8dfb998eb36353b938b41c87b6"} Oct 30 05:30:02.269325 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.269298 2581 generic.go:356] "Generic (PLEG): container finished" podID="77d685204da0cc5fa69ed201b4cfe0a4" containerID="d24f14bcbd1e4d496f361468e715510fe2777a9870a4cb50df5103ac557bf6ec" exitCode=0 Oct 30 05:30:02.269429 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.269376 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerDied","Data":"d24f14bcbd1e4d496f361468e715510fe2777a9870a4cb50df5103ac557bf6ec"} Oct 30 05:30:02.277864 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.277835 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" event={"ID":"12fe3fee-9496-4b7b-8923-d19c585cc4be","Type":"ContainerStarted","Data":"a83625753bb547479b8cfc9edc087e83052e20c023f8f582568ce55fc0a4bfb9"} Oct 30 05:30:02.330540 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.330451 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" podStartSLOduration=2.330431376 podStartE2EDuration="2.330431376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:02.289203933 +0000 UTC m=+3.580756958" watchObservedRunningTime="2026-10-30 05:30:02.330431376 +0000 UTC m=+3.621984405" Oct 30 05:30:02.871060 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.870217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:02.871060 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.870309 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:02.871060 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:02.870435 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:02.871060 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:02.870531 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret podName:56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde nodeName:}" failed. No retries permitted until 2026-10-30 05:30:04.870494923 +0000 UTC m=+6.162047930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret") pod "global-pull-secret-syncer-g6mhk" (UID: "56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:02.871060 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:02.870935 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:02.871060 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:02.871009 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs podName:671efddc-7c80-4be5-827d-0c227985c013 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:04.870992074 +0000 UTC m=+6.162545091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs") pod "network-metrics-daemon-mtw44" (UID: "671efddc-7c80-4be5-827d-0c227985c013") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:02.972045 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:02.971444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:02.972045 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:02.971624 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:02.972045 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:02.971641 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:02.972045 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:02.971653 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5c926 for pod openshift-network-diagnostics/network-check-target-bmv92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:02.972045 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:02.971706 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926 podName:0d304ba5-b889-412c-ae7d-b26f04222714 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:04.971688957 +0000 UTC m=+6.263241973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c926" (UniqueName: "kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926") pod "network-check-target-bmv92" (UID: "0d304ba5-b889-412c-ae7d-b26f04222714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:03.195674 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:03.195556 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:25:01 +0000 UTC" deadline="2028-07-09 13:04:08.209284852 +0000 UTC" Oct 30 05:30:03.195674 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:03.195595 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14839h34m5.01369423s" Oct 30 05:30:03.211324 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:03.211289 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:03.211498 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:03.211426 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:03.211857 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:03.211824 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:03.211950 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:03.211923 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:03.212040 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:03.212000 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:03.212095 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:03.212071 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:03.295637 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:03.295555 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerStarted","Data":"3dadcb6da8af20a47e583500a8ea6985c62b0bbf2f3dcd0b99584c8df7e58fc7"} Oct 30 05:30:04.885577 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:04.884715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:04.885577 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:04.884782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:04.885577 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:04.884933 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:04.885577 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:04.884997 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs podName:671efddc-7c80-4be5-827d-0c227985c013 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:08.884976736 +0000 UTC m=+10.176529754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs") pod "network-metrics-daemon-mtw44" (UID: "671efddc-7c80-4be5-827d-0c227985c013") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:04.885577 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:04.885413 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:04.885577 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:04.885485 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret podName:56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde nodeName:}" failed. No retries permitted until 2026-10-30 05:30:08.885452447 +0000 UTC m=+10.177005464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret") pod "global-pull-secret-syncer-g6mhk" (UID: "56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:04.986278 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:04.985745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:04.986278 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:04.985877 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:04.986278 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:04.985891 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:04.986278 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:04.985899 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5c926 for pod openshift-network-diagnostics/network-check-target-bmv92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:04.986278 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:04.985948 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926 podName:0d304ba5-b889-412c-ae7d-b26f04222714 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:08.985928873 +0000 UTC m=+10.277481892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c926" (UniqueName: "kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926") pod "network-check-target-bmv92" (UID: "0d304ba5-b889-412c-ae7d-b26f04222714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:05.212774 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:05.211946 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:05.212774 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:05.212073 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:05.212774 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:05.212450 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:05.212774 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:05.212566 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:05.212774 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:05.212626 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:05.212774 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:05.212691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:05.314991 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:05.312060 2581 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="ac58814d9f28f586886d10d2e6f9c2d4d06836dcd7c4b55ea499f1256bf6d8f9" exitCode=0 Oct 30 05:30:05.314991 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:05.312105 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt95j" event={"ID":"510573e7-a380-415f-9609-18470c856703","Type":"ContainerDied","Data":"ac58814d9f28f586886d10d2e6f9c2d4d06836dcd7c4b55ea499f1256bf6d8f9"} Oct 30 05:30:05.386122 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:05.385411 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" podStartSLOduration=5.385393492 podStartE2EDuration="5.385393492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:03.335605321 +0000 UTC m=+4.627158345" watchObservedRunningTime="2026-10-30 05:30:05.385393492 +0000 UTC m=+6.676946520" Oct 30 05:30:06.319772 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:06.319729 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-scqnq" event={"ID":"74f707f9-cd75-4d5f-bab3-3ead92d87904","Type":"ContainerStarted","Data":"6dba676951ac5bf29576cf42c80510bf924573cda2857d10c205ff68472ca55e"} Oct 30 05:30:07.211585 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:07.211558 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:07.211733 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:07.211680 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:07.212020 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:07.211997 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:07.212119 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:07.212096 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:07.212741 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:07.212719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:07.212833 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:07.212816 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:07.327817 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:07.327738 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vltvd" event={"ID":"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37","Type":"ContainerStarted","Data":"e2f220bb0d49b56ec82101cb066cd935af69d3799fc1666240d9295c04712cbe"} Oct 30 05:30:07.372477 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:07.369152 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-scqnq" podStartSLOduration=4.535111314 podStartE2EDuration="8.369132457s" totalImagesPullingTime="3.834021143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:06.360686465 +0000 UTC m=+7.652239494" watchObservedRunningTime="2026-10-30 05:30:07.369132457 +0000 UTC m=+8.660685481" Oct 30 05:30:08.332438 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:08.332405 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" event={"ID":"12fe3fee-9496-4b7b-8923-d19c585cc4be","Type":"ContainerStarted","Data":"8faa6f42def1d50be7e3135ad1b6b4036e99712a0527ecd919cac73235f352d9"} Oct 30 05:30:08.918895 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:08.918871 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:30:08.922269 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:08.922247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:08.922345 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:08.922284 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:08.922419 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:08.922405 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:08.922476 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:08.922451 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs podName:671efddc-7c80-4be5-827d-0c227985c013 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:16.922438767 +0000 UTC m=+18.213991769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs") pod "network-metrics-daemon-mtw44" (UID: "671efddc-7c80-4be5-827d-0c227985c013") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:08.922541 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:08.922528 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:08.922600 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:08.922575 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret podName:56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde nodeName:}" failed. No retries permitted until 2026-10-30 05:30:16.922562984 +0000 UTC m=+18.214116008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret") pod "global-pull-secret-syncer-g6mhk" (UID: "56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:09.022706 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.022630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:09.022867 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:09.022830 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:09.022867 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:09.022847 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:09.022867 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:09.022857 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5c926 for pod openshift-network-diagnostics/network-check-target-bmv92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:09.023030 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:09.022900 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926 podName:0d304ba5-b889-412c-ae7d-b26f04222714 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:17.022888074 +0000 UTC m=+18.314441089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c926" (UniqueName: "kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926") pod "network-check-target-bmv92" (UID: "0d304ba5-b889-412c-ae7d-b26f04222714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:09.042631 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.042596 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:09.043511 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.043172 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:09.086389 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.086340 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vltvd" podStartSLOduration=5.4315947 podStartE2EDuration="10.086326109s" totalImagesPullingTime="4.654731409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:07.372787281 +0000 UTC m=+8.664340307" watchObservedRunningTime="2026-10-30 05:30:09.086326109 +0000 UTC m=+10.377879184" Oct 30 05:30:09.204946 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.203992 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:30:08.918892027Z","UUID":"a2943667-577b-44da-acc9-58457f1b39a7","Handler":null,"Name":"","Endpoint":""} Oct 30 05:30:09.206412 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.206193 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:30:09.206412 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.206223 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:30:09.209210 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.209019 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:09.209210 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:09.209138 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:09.209546 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.209527 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:09.209641 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:09.209622 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:09.209712 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.209691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:09.209779 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:09.209761 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:09.337827 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.337756 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" event={"ID":"12fe3fee-9496-4b7b-8923-d19c585cc4be","Type":"ContainerStarted","Data":"bf6a3a32547aa3ca2b316c579013fb92766aa6c8924a5f4f1912538f4b7ee21f"} Oct 30 05:30:09.340327 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.340282 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jlq5d" event={"ID":"e0a3b194-c453-404f-92a4-0b1ab87f7699","Type":"ContainerStarted","Data":"51f06af63e5a30b462f05ab15db9c3ac5b8359017afcde96b9c4bd9764096775"} Oct 30 05:30:09.383903 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:09.383842 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jlq5d" podStartSLOduration=3.198506056 podStartE2EDuration="10.383823394s" totalImagesPullingTime="7.185317338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:09.383085362 +0000 UTC m=+10.674638385" watchObservedRunningTime="2026-10-30 05:30:09.383823394 +0000 UTC m=+10.675376419" Oct 30 05:30:10.343860 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:10.343602 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-trbn7" event={"ID":"f3d3980e-8842-4c87-a275-8bf0da706fba","Type":"ContainerStarted","Data":"4cf27bcbff4175a6f3e0604db1d65ae9436c9f051f1f16f32ad05e86e5004051"} Oct 30 05:30:10.346957 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:10.346925 2581 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="1511facb36682665b29767d9c47a66bc4b5b4c09373437fc90bc34a2f382cd13" exitCode=0 Oct 30 05:30:10.347109 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:10.347001 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt95j" event={"ID":"510573e7-a380-415f-9609-18470c856703","Type":"ContainerDied","Data":"1511facb36682665b29767d9c47a66bc4b5b4c09373437fc90bc34a2f382cd13"} Oct 30 05:30:10.350914 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:10.350892 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" event={"ID":"12fe3fee-9496-4b7b-8923-d19c585cc4be","Type":"ContainerStarted","Data":"98065a244109ad28a81e435d28b6707a2bb3feae6201a857cba0583e7881286d"} Oct 30 05:30:10.383865 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:10.383818 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-trbn7" podStartSLOduration=3.679773671 podStartE2EDuration="11.383807205s" totalImagesPullingTime="7.704033534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:10.383383079 +0000 UTC m=+11.674936114" watchObservedRunningTime="2026-10-30 05:30:10.383807205 +0000 UTC m=+11.675360229" Oct 30 05:30:10.474259 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:10.474197 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" podStartSLOduration=3.54662868 podStartE2EDuration="11.474183802s" totalImagesPullingTime="7.927555122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:10.425775012 +0000 UTC m=+11.717328039" watchObservedRunningTime="2026-10-30 05:30:10.474183802 +0000 UTC m=+11.765736826" Oct 30 05:30:11.068758 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.068725 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:11.068915 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.068894 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:30:11.069841 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.069822 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:11.209308 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.209232 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:11.209308 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.209256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:11.209548 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:11.209395 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:11.209603 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:11.209571 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:11.209603 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.209584 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:11.209753 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:11.209728 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:11.355513 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.355423 2581 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="4cb1284e6667520b5ee716cfd51ec96ba154f7c6544f55171b8a70b5a881d86a" exitCode=0 Oct 30 05:30:11.356006 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.355528 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt95j" event={"ID":"510573e7-a380-415f-9609-18470c856703","Type":"ContainerDied","Data":"4cb1284e6667520b5ee716cfd51ec96ba154f7c6544f55171b8a70b5a881d86a"} Oct 30 05:30:11.357183 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.357155 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-78m84" event={"ID":"9db8ada4-0aec-4b8d-844b-e7d9abbe0850","Type":"ContainerStarted","Data":"dc496c82dc95018385825278e8f58b87609ec66421f1d09f7f0994d3ca649984"} Oct 30 05:30:11.441277 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:11.441220 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-78m84" podStartSLOduration=4.866213588 podStartE2EDuration="12.441206923s" totalImagesPullingTime="7.574993335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:11.44118815 +0000 UTC m=+12.732741175" watchObservedRunningTime="2026-10-30 05:30:11.441206923 +0000 UTC m=+12.732759947" Oct 30 05:30:12.361075 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:12.360996 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" event={"ID":"62348232-2a4d-4ed2-8830-10f547a59fea","Type":"ContainerStarted","Data":"19e1feae3607e87f7d0b121103f66ced7be1192c5f281ab825d4136e94c8b5fa"} Oct 30 05:30:12.363827 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:12.363802 2581 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="1f7ff4920c9e8eaccf8f23ebcd9f0756b838d410ab6dbb707e1d36f031f967ef" exitCode=0 Oct 30 05:30:12.363956 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:12.363838 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt95j" event={"ID":"510573e7-a380-415f-9609-18470c856703","Type":"ContainerDied","Data":"1f7ff4920c9e8eaccf8f23ebcd9f0756b838d410ab6dbb707e1d36f031f967ef"} Oct 30 05:30:12.404127 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:12.404081 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" podStartSLOduration=3.3407090139999998 podStartE2EDuration="13.404069904s" totalImagesPullingTime="10.06336089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:12.403875866 +0000 UTC m=+13.695428890" watchObservedRunningTime="2026-10-30 05:30:12.404069904 +0000 UTC m=+13.695622919" Oct 30 05:30:13.209707 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:13.209671 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:13.209859 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:13.209796 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:13.210288 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:13.210260 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:13.210424 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:13.210375 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:13.210508 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:13.210424 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:13.210567 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:13.210547 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:14.372269 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:14.372082 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt95j" event={"ID":"510573e7-a380-415f-9609-18470c856703","Type":"ContainerStarted","Data":"a2b7926adcb43a8ab8c2622dc7bce4755fe7b79c6842fc93e0a3ad79f820b2ab"} Oct 30 05:30:15.209092 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:15.209062 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:15.209269 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:15.209062 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:15.209269 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:15.209188 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:15.209269 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:15.209062 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:15.209423 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:15.209284 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:15.209423 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:15.209355 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:15.376496 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:15.376441 2581 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="a2b7926adcb43a8ab8c2622dc7bce4755fe7b79c6842fc93e0a3ad79f820b2ab" exitCode=0 Oct 30 05:30:15.376496 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:15.376488 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt95j" event={"ID":"510573e7-a380-415f-9609-18470c856703","Type":"ContainerDied","Data":"a2b7926adcb43a8ab8c2622dc7bce4755fe7b79c6842fc93e0a3ad79f820b2ab"} Oct 30 05:30:16.381879 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.381843 2581 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="012aaef3e7f70d4648a128d38fa0bb440b1b5ad129eee161ec71ae6fcc51b834" exitCode=0 Oct 30 05:30:16.382432 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.381930 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt95j" event={"ID":"510573e7-a380-415f-9609-18470c856703","Type":"ContainerDied","Data":"012aaef3e7f70d4648a128d38fa0bb440b1b5ad129eee161ec71ae6fcc51b834"} Oct 30 05:30:16.386784 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.386576 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" event={"ID":"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3","Type":"ContainerStarted","Data":"abc65c05d75c4f31da36de416e12d8a2998fa41a5008cb0a7565b52ab540e1b1"} Oct 30 05:30:16.386875 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.386843 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" event={"ID":"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3","Type":"ContainerStarted","Data":"cb6c20cb29a4892c9c40ae471ee2031d286ea31dd8059770d9c8194be015063a"} Oct 30 05:30:16.386916 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.386890 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" event={"ID":"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3","Type":"ContainerStarted","Data":"aeddcf29a4ba9637e2028ef4a2e390a52b6fbf0f1bfce217d3d3178ac036f0f6"} Oct 30 05:30:16.386948 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.386932 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" event={"ID":"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3","Type":"ContainerStarted","Data":"3c88e17e7fac72bd690317cf4b9ec52db7c363b8277e1559a63571e55519ab5a"} Oct 30 05:30:16.386994 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.386972 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" event={"ID":"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3","Type":"ContainerStarted","Data":"90bc3c0e8cf17db114f7b0d1371095b7d4437ee0b765e696532a5eaba69b59d2"} Oct 30 05:30:16.387030 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.387017 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" event={"ID":"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3","Type":"ContainerStarted","Data":"295ad61d48d05db3b0b62cbbb2983545f1c5d4622ec921283511fe366056a0e5"} Oct 30 05:30:16.984663 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.984628 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:16.984821 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:16.984693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:16.984821 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:16.984773 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:16.984821 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:16.984772 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:16.984821 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:16.984822 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret podName:56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.984809644 +0000 UTC m=+34.276362647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret") pod "global-pull-secret-syncer-g6mhk" (UID: "56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:16.984957 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:16.984835 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs podName:671efddc-7c80-4be5-827d-0c227985c013 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.984828908 +0000 UTC m=+34.276381914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs") pod "network-metrics-daemon-mtw44" (UID: "671efddc-7c80-4be5-827d-0c227985c013") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:17.085758 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:17.085720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:17.085901 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:17.085856 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:17.085901 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:17.085872 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:17.085901 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:17.085880 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5c926 for pod openshift-network-diagnostics/network-check-target-bmv92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:17.085994 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:17.085932 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926 podName:0d304ba5-b889-412c-ae7d-b26f04222714 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:33.08591705 +0000 UTC m=+34.377470058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c926" (UniqueName: "kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926") pod "network-check-target-bmv92" (UID: "0d304ba5-b889-412c-ae7d-b26f04222714") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:17.212542 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:17.212519 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:17.212691 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:17.212548 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:17.212691 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:17.212620 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:17.212756 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:17.212679 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:17.212756 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:17.212712 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:17.212819 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:17.212775 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:17.393824 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:17.393740 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt95j" event={"ID":"510573e7-a380-415f-9609-18470c856703","Type":"ContainerStarted","Data":"97e71ff2c04e5b41f4a1dbc62f91911c6a5cc9f0ca72715e0a6ee473932b3f65"} Oct 30 05:30:17.441403 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:17.441363 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kt95j" podStartSLOduration=8.338574241 podStartE2EDuration="18.441352417s" totalImagesPullingTime="10.102778176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:17.44116344 +0000 UTC m=+18.732716464" watchObservedRunningTime="2026-10-30 05:30:17.441352417 +0000 UTC m=+18.732905441" Oct 30 05:30:18.398843 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:18.398764 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" event={"ID":"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3","Type":"ContainerStarted","Data":"fb2c1257e22ec335efaa38431488bb4a577ee9f4bb49f22c8ef0493665edc323"} Oct 30 05:30:19.209769 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:19.209739 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:19.209942 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:19.209886 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:19.209942 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:19.209900 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:19.210061 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:19.210015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:19.210115 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:19.210058 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:19.210115 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:19.210092 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:20.406507 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:20.406415 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" event={"ID":"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3","Type":"ContainerStarted","Data":"c0134128b61edf7d611d250d681886c70814e5ac38608fc667371ab70ae2fd77"} Oct 30 05:30:20.407139 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:20.406673 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:20.422542 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:20.422515 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:20.496531 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:20.496487 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" podStartSLOduration=7.341691155 podStartE2EDuration="21.496475825s" totalImagesPullingTime="14.15478467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:20.484014519 +0000 UTC m=+21.775567568" watchObservedRunningTime="2026-10-30 05:30:20.496475825 +0000 UTC m=+21.788028840" Oct 30 05:30:21.209383 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:21.209182 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:21.209569 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:21.209205 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:21.209569 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:21.209485 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:21.209569 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:21.209552 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:21.209569 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:21.209228 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:21.209719 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:21.209640 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:21.408897 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:21.408872 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:21.409255 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:21.408945 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:21.424071 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:21.424049 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:22.199169 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:22.199143 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g6mhk"] Oct 30 05:30:22.199373 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:22.199236 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:22.199373 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:22.199317 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:22.201938 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:22.201917 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bmv92"] Oct 30 05:30:22.202047 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:22.201988 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:22.202083 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:22.202051 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:22.204979 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:22.204958 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mtw44"] Oct 30 05:30:22.205085 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:22.205035 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:22.205124 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:22.205105 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:24.209728 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:24.209698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:24.210165 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:24.209763 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:24.210165 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:24.209786 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:24.210165 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:24.209871 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:24.210165 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:24.209910 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:24.210165 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:24.209979 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:26.208920 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:26.208881 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:26.209345 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:26.208922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:26.209345 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:26.208986 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:26.209345 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:26.209006 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:26.209345 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:26.209102 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:26.209345 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:26.209168 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:28.209526 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.209502 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:28.209852 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.209533 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:28.209852 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.209510 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:28.209852 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.209605 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtw44" podUID="671efddc-7c80-4be5-827d-0c227985c013" Oct 30 05:30:28.209852 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.209676 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g6mhk" podUID="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" Oct 30 05:30:28.209852 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.209727 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmv92" podUID="0d304ba5-b889-412c-ae7d-b26f04222714" Oct 30 05:30:28.474777 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.474704 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeReady" Oct 30 05:30:28.474920 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.474818 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 05:30:28.522841 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.522806 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4fnpv"] Oct 30 05:30:28.526234 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.526212 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l5k87"] Oct 30 05:30:28.526362 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.526341 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" Oct 30 05:30:28.528992 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.528974 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kb9n4"] Oct 30 05:30:28.529125 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.529106 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l5k87" Oct 30 05:30:28.531734 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.531717 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wr9wr"] Oct 30 05:30:28.531821 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.531808 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" Oct 30 05:30:28.535044 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.535028 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr"] Oct 30 05:30:28.535154 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.535139 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.537967 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.537948 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cc48cb9d8-jzp2t"] Oct 30 05:30:28.538592 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.538569 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.547160 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.547140 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.547160 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.547154 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.553729 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.553710 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 05:30:28.553900 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.553783 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 05:30:28.554081 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.554064 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2m9cf\"" Oct 30 05:30:28.554199 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.554177 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hc6rm\"" Oct 30 05:30:28.554591 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.554477 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-869865c857-td99r"] Oct 30 05:30:28.554591 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.554586 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.562391 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.562354 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj"] Oct 30 05:30:28.563359 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.562617 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.575879 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.575856 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn"] Oct 30 05:30:28.576114 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.575964 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" Oct 30 05:30:28.579775 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.579756 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d"] Oct 30 05:30:28.580526 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.579949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.582982 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.582966 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-c9p4j"] Oct 30 05:30:28.583314 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.583293 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:28.587653 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.587636 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw"] Oct 30 05:30:28.588441 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.588415 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.591796 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.591772 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-54dbcc76d9-x7c5h"] Oct 30 05:30:28.592149 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.592127 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:28.595865 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.595846 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-l725b"] Oct 30 05:30:28.596024 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.596012 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.601663 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.601646 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-846hg"] Oct 30 05:30:28.601777 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.601763 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:28.605343 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.605326 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bnfpg"] Oct 30 05:30:28.605568 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.605550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" Oct 30 05:30:28.608567 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.608550 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x2s8x"] Oct 30 05:30:28.608771 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.608751 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.612174 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612155 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4fnpv"] Oct 30 05:30:28.612255 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612202 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l5k87"] Oct 30 05:30:28.612255 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612226 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wr9wr"] Oct 30 05:30:28.612255 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612250 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr"] Oct 30 05:30:28.612335 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612262 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:28.612335 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612271 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn"] Oct 30 05:30:28.612335 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612295 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc48cb9d8-jzp2t"] Oct 30 05:30:28.612335 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612315 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612340 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612372 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869865c857-td99r"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612400 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bnfpg"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612425 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x2s8x"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612445 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kb9n4"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612516 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-846hg"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612544 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-c9p4j"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612585 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54dbcc76d9-x7c5h"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612610 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-l725b"] Oct 30 05:30:28.612958 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.612629 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj"] Oct 30 05:30:28.647095 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647073 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.647205 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647128 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.647205 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647138 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.647324 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647233 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 05:30:28.647509 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647447 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 05:30:28.647509 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647503 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.647749 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647732 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.647817 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647771 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.647817 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647780 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.647919 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647856 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.647919 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647905 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.648015 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647935 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 05:30:28.648015 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.647976 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.648112 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648077 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.648302 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648290 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.648357 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648339 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.648502 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648439 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 05:30:28.648502 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648483 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.648885 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648521 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.648885 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648291 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.648885 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648550 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.648885 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648519 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 05:30:28.648885 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648591 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.648885 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.648609 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 05:30:28.649189 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.649168 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:30:28.649307 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.649217 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:28.649385 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.649176 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:28.649385 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.649373 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 05:30:28.669245 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669225 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n589\" (UniqueName: \"kubernetes.io/projected/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-kube-api-access-5n589\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.669336 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669265 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.669336 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669288 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.669401 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.669430 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkzgj\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.669488 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.669527 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669506 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.669573 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:28.669625 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669571 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4wgj\" (UniqueName: \"kubernetes.io/projected/6381b6d8-a45f-43d3-955c-504694db87da-kube-api-access-p4wgj\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.669671 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669631 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxd2m\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.669721 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669672 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.669758 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669723 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.669808 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669757 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.669857 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669805 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-tmp\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.669857 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669835 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/afe5578f-4885-4a55-a947-4c9bfe331614-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:28.669938 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d813b7e-2c08-4c0f-b077-2b529d90cf51-serving-cert\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.669938 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669908 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.670029 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-snapshots\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.670029 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.669996 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-service-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.670126 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f968039-9ce3-48ea-a1c6-c963206a596c-config\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.670126 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670064 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-trusted-ca\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.670348 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7khr\" (UniqueName: \"kubernetes.io/projected/f25962bc-3b24-4f15-842f-e3d9771a1809-kube-api-access-g7khr\") pod \"kas-connection-checker-685895c7c9-kb9n4\" (UID: \"f25962bc-3b24-4f15-842f-e3d9771a1809\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" Oct 30 05:30:28.670348 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670332 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.670450 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670359 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.670450 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xddvn\" (UniqueName: \"kubernetes.io/projected/d0f89396-3754-4034-af10-b9a951d3bbb1-kube-api-access-xddvn\") pod \"volume-data-source-validator-58c9f5f46d-dggnj\" (UID: \"d0f89396-3754-4034-af10-b9a951d3bbb1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" Oct 30 05:30:28.670450 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670426 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnkmt\" (UniqueName: \"kubernetes.io/projected/60d71bc3-a236-4305-baeb-df6403b52f43-kube-api-access-lnkmt\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.670639 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670479 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.670639 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670507 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-stats-auth\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.670639 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfrm\" (UniqueName: \"kubernetes.io/projected/afe5578f-4885-4a55-a947-4c9bfe331614-kube-api-access-wmfrm\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:28.670639 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zrt\" (UniqueName: \"kubernetes.io/projected/ae0a00ed-126f-4c35-a159-aa6ed7e2c72f-kube-api-access-h5zrt\") pod \"kas-connection-checker-685895c7c9-4fnpv\" (UID: \"ae0a00ed-126f-4c35-a159-aa6ed7e2c72f\") " pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" Oct 30 05:30:28.670783 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.670783 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670659 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.670783 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670677 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgr9\" (UniqueName: \"kubernetes.io/projected/2f968039-9ce3-48ea-a1c6-c963206a596c-kube-api-access-bqgr9\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.670783 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.670783 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670724 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60d71bc3-a236-4305-baeb-df6403b52f43-serving-cert\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.670783 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1e52c123-4401-4329-8058-5479f3680281-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:28.671080 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670818 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-config\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.671080 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-default-certificate\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.671080 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670934 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.671080 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.670968 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.671080 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.671040 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f968039-9ce3-48ea-a1c6-c963206a596c-serving-cert\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.671080 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.671075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:28.671365 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.671099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.671365 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.671134 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.671365 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.671178 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqgpk\" (UniqueName: \"kubernetes.io/projected/1e52c123-4401-4329-8058-5479f3680281-kube-api-access-kqgpk\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:28.671365 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.671233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkvb\" (UniqueName: \"kubernetes.io/projected/8d813b7e-2c08-4c0f-b077-2b529d90cf51-kube-api-access-5kkvb\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.671365 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.671279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjx5n\" (UniqueName: \"kubernetes.io/projected/9e2abeba-b181-40af-89c2-c3782ad160c5-kube-api-access-vjx5n\") pod \"kas-connection-checker-685895c7c9-l5k87\" (UID: \"9e2abeba-b181-40af-89c2-c3782ad160c5\") " pod="kube-system/kas-connection-checker-685895c7c9-l5k87" Oct 30 05:30:28.673258 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673238 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 05:30:28.673365 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673350 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 05:30:28.673427 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673403 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 05:30:28.673598 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673583 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t4r8b\"" Oct 30 05:30:28.673693 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673676 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 05:30:28.673777 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673723 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:30:28.673777 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673726 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 05:30:28.673777 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673745 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 05:30:28.673921 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673781 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 05:30:28.673921 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673783 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 05:30:28.673921 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673796 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 05:30:28.673921 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673801 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9q7xl\"" Oct 30 05:30:28.673921 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673891 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:30:28.673921 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673910 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673927 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673933 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bfw5z\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673974 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.673984 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vgfqj\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.674023 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-w2lf4\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.674059 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.674099 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-n79jw\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.674101 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r79mb\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.674159 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9nrzf\"" Oct 30 05:30:28.674192 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.674185 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 05:30:28.674627 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.674233 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 05:30:28.674627 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.674237 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-49l47\"" Oct 30 05:30:28.674627 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.674261 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wzzmx\"" Oct 30 05:30:28.772624 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zrt\" (UniqueName: \"kubernetes.io/projected/ae0a00ed-126f-4c35-a159-aa6ed7e2c72f-kube-api-access-h5zrt\") pod \"kas-connection-checker-685895c7c9-4fnpv\" (UID: \"ae0a00ed-126f-4c35-a159-aa6ed7e2c72f\") " pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" Oct 30 05:30:28.772624 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772598 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.772804 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772626 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.772804 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgr9\" (UniqueName: \"kubernetes.io/projected/2f968039-9ce3-48ea-a1c6-c963206a596c-kube-api-access-bqgr9\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.772804 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772684 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.772804 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772707 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60d71bc3-a236-4305-baeb-df6403b52f43-serving-cert\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.772804 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1e52c123-4401-4329-8058-5479f3680281-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:28.772804 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-config\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.772804 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/429c555a-f735-446f-8577-4f7670a5d094-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772825 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-default-certificate\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.772884 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:29.272850424 +0000 UTC m=+30.564403466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : configmap references non-existent config key: service-ca.crt Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.772968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f968039-9ce3-48ea-a1c6-c963206a596c-serving-cert\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773033 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee3f1bf-665e-4220-ac90-5d0b041d7077-config-volume\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.773141 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773118 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqgpk\" (UniqueName: \"kubernetes.io/projected/1e52c123-4401-4329-8058-5479f3680281-kube-api-access-kqgpk\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773146 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkvb\" (UniqueName: \"kubernetes.io/projected/8d813b7e-2c08-4c0f-b077-2b529d90cf51-kube-api-access-5kkvb\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjx5n\" (UniqueName: \"kubernetes.io/projected/9e2abeba-b181-40af-89c2-c3782ad160c5-kube-api-access-vjx5n\") pod \"kas-connection-checker-685895c7c9-l5k87\" (UID: \"9e2abeba-b181-40af-89c2-c3782ad160c5\") " pod="kube-system/kas-connection-checker-685895c7c9-l5k87" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773245 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wj77\" (UniqueName: \"kubernetes.io/projected/26ef5d7f-4915-4cee-a412-c1f1f38410bd-kube-api-access-8wj77\") pod \"network-check-source-6f78d59f99-846hg\" (UID: \"26ef5d7f-4915-4cee-a412-c1f1f38410bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n589\" (UniqueName: \"kubernetes.io/projected/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-kube-api-access-5n589\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773321 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773346 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773400 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzgj\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ts77\" (UniqueName: \"kubernetes.io/projected/0ee3f1bf-665e-4220-ac90-5d0b041d7077-kube-api-access-6ts77\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773447 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773543 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-config\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773595 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1e52c123-4401-4329-8058-5479f3680281-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:28.773684 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.773622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:28.774381 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.773702 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:30:28.774381 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.773773 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls podName:1e52c123-4401-4329-8058-5479f3680281 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:29.27375496 +0000 UTC m=+30.565307976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gx2pw" (UID: "1e52c123-4401-4329-8058-5479f3680281") : secret "samples-operator-tls" not found Oct 30 05:30:28.774676 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.774657 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.774789 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.774773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgps8\" (UniqueName: \"kubernetes.io/projected/540c21f9-43da-400c-b0da-e0b01eecedf7-kube-api-access-dgps8\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:28.774849 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.774813 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4wgj\" (UniqueName: \"kubernetes.io/projected/6381b6d8-a45f-43d3-955c-504694db87da-kube-api-access-p4wgj\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.774849 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.774842 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxd2m\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.774952 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.774871 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.774952 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.774901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.774952 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.774937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.775094 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.774979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.775094 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775007 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-tmp\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.775094 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f1bf-665e-4220-ac90-5d0b041d7077-tmp-dir\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.775094 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/afe5578f-4885-4a55-a947-4c9bfe331614-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:28.775094 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775084 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d813b7e-2c08-4c0f-b077-2b529d90cf51-serving-cert\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.775342 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775115 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.775342 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-snapshots\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.775342 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-service-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.775342 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775201 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:28.775342 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f968039-9ce3-48ea-a1c6-c963206a596c-config\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.775342 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-trusted-ca\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.775342 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775294 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7khr\" (UniqueName: \"kubernetes.io/projected/f25962bc-3b24-4f15-842f-e3d9771a1809-kube-api-access-g7khr\") pod \"kas-connection-checker-685895c7c9-kb9n4\" (UID: \"f25962bc-3b24-4f15-842f-e3d9771a1809\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" Oct 30 05:30:28.775719 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-tmp\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.775719 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.775719 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.775864 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddvn\" (UniqueName: \"kubernetes.io/projected/d0f89396-3754-4034-af10-b9a951d3bbb1-kube-api-access-xddvn\") pod \"volume-data-source-validator-58c9f5f46d-dggnj\" (UID: \"d0f89396-3754-4034-af10-b9a951d3bbb1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" Oct 30 05:30:28.775864 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775758 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkmt\" (UniqueName: \"kubernetes.io/projected/60d71bc3-a236-4305-baeb-df6403b52f43-kube-api-access-lnkmt\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.775864 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775766 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.775864 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775786 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/540c21f9-43da-400c-b0da-e0b01eecedf7-cert\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:28.775864 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.775864 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-stats-auth\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.775864 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.775857 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:28.776233 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.775880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfrm\" (UniqueName: \"kubernetes.io/projected/afe5578f-4885-4a55-a947-4c9bfe331614-kube-api-access-wmfrm\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:28.776233 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.775908 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls podName:afe5578f-4885-4a55-a947-4c9bfe331614 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:29.275893147 +0000 UTC m=+30.567446149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4mn4d" (UID: "afe5578f-4885-4a55-a947-4c9bfe331614") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:28.776233 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.775968 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:28.776233 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.775980 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc48cb9d8-jzp2t: secret "image-registry-tls" not found Oct 30 05:30:28.776233 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.776012 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls podName:35863a55-5422-4304-ad71-afddab4d4807 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:29.275999978 +0000 UTC m=+30.567552994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls") pod "image-registry-cc48cb9d8-jzp2t" (UID: "35863a55-5422-4304-ad71-afddab4d4807") : secret "image-registry-tls" not found Oct 30 05:30:28.777062 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.777037 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.777160 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.777059 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/afe5578f-4885-4a55-a947-4c9bfe331614-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:28.777223 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.776995 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.777495 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.777474 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f968039-9ce3-48ea-a1c6-c963206a596c-config\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.778430 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.777505 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:28.778568 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.778555 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869865c857-td99r: secret "image-registry-tls" not found Oct 30 05:30:28.778691 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.778679 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls podName:2fd827f0-3153-4b94-8326-8c6fd0744f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:29.278661185 +0000 UTC m=+30.570214203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls") pod "image-registry-869865c857-td99r" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b") : secret "image-registry-tls" not found Oct 30 05:30:28.778830 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.777871 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:30:28.778892 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.778861 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:29.278847739 +0000 UTC m=+30.570400755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : secret "router-metrics-certs-default" not found Oct 30 05:30:28.778892 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.778564 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.778892 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.778298 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-trusted-ca\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.778892 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.777783 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-snapshots\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.779115 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.778765 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60d71bc3-a236-4305-baeb-df6403b52f43-serving-cert\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.779115 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.778200 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-service-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.779115 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.778808 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.779115 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.778047 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.779115 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.778862 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.779383 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.779113 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-default-certificate\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.779383 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.779297 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.779383 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.779321 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f968039-9ce3-48ea-a1c6-c963206a596c-serving-cert\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.780118 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.780100 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.780386 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.780364 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d813b7e-2c08-4c0f-b077-2b529d90cf51-serving-cert\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.780931 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.780904 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-stats-auth\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.781182 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.781165 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.781372 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.781356 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.796764 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.796746 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zrt\" (UniqueName: \"kubernetes.io/projected/ae0a00ed-126f-4c35-a159-aa6ed7e2c72f-kube-api-access-h5zrt\") pod \"kas-connection-checker-685895c7c9-4fnpv\" (UID: \"ae0a00ed-126f-4c35-a159-aa6ed7e2c72f\") " pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" Oct 30 05:30:28.796910 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.796892 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.796971 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.796913 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgr9\" (UniqueName: \"kubernetes.io/projected/2f968039-9ce3-48ea-a1c6-c963206a596c-kube-api-access-bqgr9\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.798762 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.798743 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7khr\" (UniqueName: \"kubernetes.io/projected/f25962bc-3b24-4f15-842f-e3d9771a1809-kube-api-access-g7khr\") pod \"kas-connection-checker-685895c7c9-kb9n4\" (UID: \"f25962bc-3b24-4f15-842f-e3d9771a1809\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" Oct 30 05:30:28.800552 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.800530 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfrm\" (UniqueName: \"kubernetes.io/projected/afe5578f-4885-4a55-a947-4c9bfe331614-kube-api-access-wmfrm\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:28.801385 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.801362 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxd2m\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.801700 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.801661 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4wgj\" (UniqueName: \"kubernetes.io/projected/6381b6d8-a45f-43d3-955c-504694db87da-kube-api-access-p4wgj\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:28.802840 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.802816 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqgpk\" (UniqueName: \"kubernetes.io/projected/1e52c123-4401-4329-8058-5479f3680281-kube-api-access-kqgpk\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:28.804705 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.804679 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkvb\" (UniqueName: \"kubernetes.io/projected/8d813b7e-2c08-4c0f-b077-2b529d90cf51-kube-api-access-5kkvb\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.805106 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.805040 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddvn\" (UniqueName: \"kubernetes.io/projected/d0f89396-3754-4034-af10-b9a951d3bbb1-kube-api-access-xddvn\") pod \"volume-data-source-validator-58c9f5f46d-dggnj\" (UID: \"d0f89396-3754-4034-af10-b9a951d3bbb1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" Oct 30 05:30:28.805513 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.805488 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n589\" (UniqueName: \"kubernetes.io/projected/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-kube-api-access-5n589\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.806724 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.806586 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzgj\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:28.807145 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.807119 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnkmt\" (UniqueName: \"kubernetes.io/projected/60d71bc3-a236-4305-baeb-df6403b52f43-kube-api-access-lnkmt\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.807235 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.807204 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:28.807670 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.807650 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjx5n\" (UniqueName: \"kubernetes.io/projected/9e2abeba-b181-40af-89c2-c3782ad160c5-kube-api-access-vjx5n\") pod \"kas-connection-checker-685895c7c9-l5k87\" (UID: \"9e2abeba-b181-40af-89c2-c3782ad160c5\") " pod="kube-system/kas-connection-checker-685895c7c9-l5k87" Oct 30 05:30:28.851095 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.851067 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" Oct 30 05:30:28.854129 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.854109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l5k87" Oct 30 05:30:28.857952 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.857929 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" Oct 30 05:30:28.875014 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.874985 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:28.876280 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.876254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgps8\" (UniqueName: \"kubernetes.io/projected/540c21f9-43da-400c-b0da-e0b01eecedf7-kube-api-access-dgps8\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:28.876381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.876298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.876381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.876330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f1bf-665e-4220-ac90-5d0b041d7077-tmp-dir\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.876381 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.876358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:28.876577 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.876385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/540c21f9-43da-400c-b0da-e0b01eecedf7-cert\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:28.876577 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.876447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/429c555a-f735-446f-8577-4f7670a5d094-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:28.876577 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.876504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee3f1bf-665e-4220-ac90-5d0b041d7077-config-volume\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.876577 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.876534 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wj77\" (UniqueName: \"kubernetes.io/projected/26ef5d7f-4915-4cee-a412-c1f1f38410bd-kube-api-access-8wj77\") pod \"network-check-source-6f78d59f99-846hg\" (UID: \"26ef5d7f-4915-4cee-a412-c1f1f38410bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" Oct 30 05:30:28.876577 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.876575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ts77\" (UniqueName: \"kubernetes.io/projected/0ee3f1bf-665e-4220-ac90-5d0b041d7077-kube-api-access-6ts77\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.877021 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.877006 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:30:28.877108 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.877053 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls podName:0ee3f1bf-665e-4220-ac90-5d0b041d7077 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:29.37704015 +0000 UTC m=+30.668593152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls") pod "dns-default-bnfpg" (UID: "0ee3f1bf-665e-4220-ac90-5d0b041d7077") : secret "dns-default-metrics-tls" not found Oct 30 05:30:28.877368 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.877348 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f1bf-665e-4220-ac90-5d0b041d7077-tmp-dir\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.877427 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.877417 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:30:28.877505 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:28.877448 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert podName:429c555a-f735-446f-8577-4f7670a5d094 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:29.377438321 +0000 UTC m=+30.668991336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l725b" (UID: "429c555a-f735-446f-8577-4f7670a5d094") : secret "networking-console-plugin-cert" not found Oct 30 05:30:28.878164 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.878118 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee3f1bf-665e-4220-ac90-5d0b041d7077-config-volume\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.878367 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.878347 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/429c555a-f735-446f-8577-4f7670a5d094-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:28.882183 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.882158 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/540c21f9-43da-400c-b0da-e0b01eecedf7-cert\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:28.886997 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.886974 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:28.903811 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.903751 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ts77\" (UniqueName: \"kubernetes.io/projected/0ee3f1bf-665e-4220-ac90-5d0b041d7077-kube-api-access-6ts77\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:28.909449 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.909363 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wj77\" (UniqueName: \"kubernetes.io/projected/26ef5d7f-4915-4cee-a412-c1f1f38410bd-kube-api-access-8wj77\") pod \"network-check-source-6f78d59f99-846hg\" (UID: \"26ef5d7f-4915-4cee-a412-c1f1f38410bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" Oct 30 05:30:28.910604 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.910347 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" Oct 30 05:30:28.910858 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.910835 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgps8\" (UniqueName: \"kubernetes.io/projected/540c21f9-43da-400c-b0da-e0b01eecedf7-kube-api-access-dgps8\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:28.924594 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.924353 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:28.939105 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.937249 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:28.958485 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:28.957233 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" Oct 30 05:30:29.003575 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.003354 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:29.189685 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.189660 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4fnpv"] Oct 30 05:30:29.190567 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:29.190541 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0a00ed_126f_4c35_a159_aa6ed7e2c72f.slice/crio-00fddd0a2156689a45838a017d5d90848112f2e7b6ff991cee5c446abeb22b98 WatchSource:0}: Error finding container 00fddd0a2156689a45838a017d5d90848112f2e7b6ff991cee5c446abeb22b98: Status 404 returned error can't find the container with id 00fddd0a2156689a45838a017d5d90848112f2e7b6ff991cee5c446abeb22b98 Oct 30 05:30:29.192747 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.192599 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kb9n4"] Oct 30 05:30:29.193653 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:29.193624 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25962bc_3b24_4f15_842f_e3d9771a1809.slice/crio-02c6460b7041ae4d79c7f5dbf6fbbd60a71d9713ffe867c73db631f1284b457a WatchSource:0}: Error finding container 02c6460b7041ae4d79c7f5dbf6fbbd60a71d9713ffe867c73db631f1284b457a: Status 404 returned error can't find the container with id 02c6460b7041ae4d79c7f5dbf6fbbd60a71d9713ffe867c73db631f1284b457a Oct 30 05:30:29.194239 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.194210 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l5k87"] Oct 30 05:30:29.194373 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:29.194350 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2abeba_b181_40af_89c2_c3782ad160c5.slice/crio-4aa810268db76f323f7ed1ed6135c0d43b257fe72cd6acc3d45d57412a328952 WatchSource:0}: Error finding container 4aa810268db76f323f7ed1ed6135c0d43b257fe72cd6acc3d45d57412a328952: Status 404 returned error can't find the container with id 4aa810268db76f323f7ed1ed6135c0d43b257fe72cd6acc3d45d57412a328952 Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.280338 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.280402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.280448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.280494 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.280548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.280584 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.280754 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.280812 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls podName:1e52c123-4401-4329-8058-5479f3680281 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:30.280793495 +0000 UTC m=+31.572346510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gx2pw" (UID: "1e52c123-4401-4329-8058-5479f3680281") : secret "samples-operator-tls" not found Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281178 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281231 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:30.281215931 +0000 UTC m=+31.572768948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : secret "router-metrics-certs-default" not found Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281300 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:30.281290039 +0000 UTC m=+31.572843058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : configmap references non-existent config key: service-ca.crt Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281360 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281395 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls podName:afe5578f-4885-4a55-a947-4c9bfe331614 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:30.281384272 +0000 UTC m=+31.572937289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4mn4d" (UID: "afe5578f-4885-4a55-a947-4c9bfe331614") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281451 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:29.281686 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281483 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc48cb9d8-jzp2t: secret "image-registry-tls" not found Oct 30 05:30:29.282822 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281514 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls podName:35863a55-5422-4304-ad71-afddab4d4807 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:30.281503783 +0000 UTC m=+31.573056801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls") pod "image-registry-cc48cb9d8-jzp2t" (UID: "35863a55-5422-4304-ad71-afddab4d4807") : secret "image-registry-tls" not found Oct 30 05:30:29.282822 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281572 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:29.282822 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281581 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869865c857-td99r: secret "image-registry-tls" not found Oct 30 05:30:29.282822 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.281609 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls podName:2fd827f0-3153-4b94-8326-8c6fd0744f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:30.281599368 +0000 UTC m=+31.573152387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls") pod "image-registry-869865c857-td99r" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b") : secret "image-registry-tls" not found Oct 30 05:30:29.286674 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.286231 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj"] Oct 30 05:30:29.289904 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.289883 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr"] Oct 30 05:30:29.289968 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.289938 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wr9wr"] Oct 30 05:30:29.292775 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:29.292532 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f89396_3754_4034_af10_b9a951d3bbb1.slice/crio-5c12fb2a72906e947586951e95e8e27f96c1dd35f6c9071342448307434a61e1 WatchSource:0}: Error finding container 5c12fb2a72906e947586951e95e8e27f96c1dd35f6c9071342448307434a61e1: Status 404 returned error can't find the container with id 5c12fb2a72906e947586951e95e8e27f96c1dd35f6c9071342448307434a61e1 Oct 30 05:30:29.293243 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:29.293220 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d71bc3_a236_4305_baeb_df6403b52f43.slice/crio-3efc823edb824143ff5be061d45040dbb07f3f226cc5dfb01fc07c77e286852b WatchSource:0}: Error finding container 3efc823edb824143ff5be061d45040dbb07f3f226cc5dfb01fc07c77e286852b: Status 404 returned error can't find the container with id 3efc823edb824143ff5be061d45040dbb07f3f226cc5dfb01fc07c77e286852b Oct 30 05:30:29.346428 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.346401 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-c9p4j"] Oct 30 05:30:29.346906 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:29.346885 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d813b7e_2c08_4c0f_b077_2b529d90cf51.slice/crio-33f15deada5b62f8a545c185d95583be81f72bf5816b9ac2fc84ce6433ca2153 WatchSource:0}: Error finding container 33f15deada5b62f8a545c185d95583be81f72bf5816b9ac2fc84ce6433ca2153: Status 404 returned error can't find the container with id 33f15deada5b62f8a545c185d95583be81f72bf5816b9ac2fc84ce6433ca2153 Oct 30 05:30:29.349248 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.348994 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn"] Oct 30 05:30:29.351790 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:29.351763 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f968039_9ce3_48ea_a1c6_c963206a596c.slice/crio-3ef490aca0227af7a5d71c0e102ab90e5c89dc957fe943726430d855908c8893 WatchSource:0}: Error finding container 3ef490aca0227af7a5d71c0e102ab90e5c89dc957fe943726430d855908c8893: Status 404 returned error can't find the container with id 3ef490aca0227af7a5d71c0e102ab90e5c89dc957fe943726430d855908c8893 Oct 30 05:30:29.353579 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.353559 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-846hg"] Oct 30 05:30:29.356148 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.356127 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x2s8x"] Oct 30 05:30:29.356927 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:29.356904 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ef5d7f_4915_4cee_a412_c1f1f38410bd.slice/crio-628153e9c7bd40d1735c99a48b322270297393fa436beba06023c088c07c9b4c WatchSource:0}: Error finding container 628153e9c7bd40d1735c99a48b322270297393fa436beba06023c088c07c9b4c: Status 404 returned error can't find the container with id 628153e9c7bd40d1735c99a48b322270297393fa436beba06023c088c07c9b4c Oct 30 05:30:29.357698 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:29.357679 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540c21f9_43da_400c_b0da_e0b01eecedf7.slice/crio-0e41d4c791823b5ac233563e3df10bd5abf5b884b016d0c77179feafaf7d1db9 WatchSource:0}: Error finding container 0e41d4c791823b5ac233563e3df10bd5abf5b884b016d0c77179feafaf7d1db9: Status 404 returned error can't find the container with id 0e41d4c791823b5ac233563e3df10bd5abf5b884b016d0c77179feafaf7d1db9 Oct 30 05:30:29.381531 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.381508 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:29.381627 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.381575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:29.381794 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.381779 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:30:29.381883 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.381848 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert podName:429c555a-f735-446f-8577-4f7670a5d094 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:30.381820262 +0000 UTC m=+31.673373283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l725b" (UID: "429c555a-f735-446f-8577-4f7670a5d094") : secret "networking-console-plugin-cert" not found Oct 30 05:30:29.381955 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.381931 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:30:29.382051 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:29.381967 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls podName:0ee3f1bf-665e-4220-ac90-5d0b041d7077 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:30.381955497 +0000 UTC m=+31.673508502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls") pod "dns-default-bnfpg" (UID: "0ee3f1bf-665e-4220-ac90-5d0b041d7077") : secret "dns-default-metrics-tls" not found Oct 30 05:30:29.425672 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.425641 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" event={"ID":"2f968039-9ce3-48ea-a1c6-c963206a596c","Type":"ContainerStarted","Data":"3ef490aca0227af7a5d71c0e102ab90e5c89dc957fe943726430d855908c8893"} Oct 30 05:30:29.426883 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.426840 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" event={"ID":"d0f89396-3754-4034-af10-b9a951d3bbb1","Type":"ContainerStarted","Data":"5c12fb2a72906e947586951e95e8e27f96c1dd35f6c9071342448307434a61e1"} Oct 30 05:30:29.428213 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.428174 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" event={"ID":"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d","Type":"ContainerStarted","Data":"95b5dd0ac769e0a1c9936ffb419279b0ae84622b3c2a5678de83b2d9274d7a2e"} Oct 30 05:30:29.430021 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.430000 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" event={"ID":"f25962bc-3b24-4f15-842f-e3d9771a1809","Type":"ContainerStarted","Data":"d2e77003df71ef1fa8d3bd926d9516340a0f076764290d308a53de85f5a49dde"} Oct 30 05:30:29.430108 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.430043 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" event={"ID":"f25962bc-3b24-4f15-842f-e3d9771a1809","Type":"ContainerStarted","Data":"02c6460b7041ae4d79c7f5dbf6fbbd60a71d9713ffe867c73db631f1284b457a"} Oct 30 05:30:29.431157 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.431139 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x2s8x" event={"ID":"540c21f9-43da-400c-b0da-e0b01eecedf7","Type":"ContainerStarted","Data":"0e41d4c791823b5ac233563e3df10bd5abf5b884b016d0c77179feafaf7d1db9"} Oct 30 05:30:29.432143 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.432127 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" event={"ID":"26ef5d7f-4915-4cee-a412-c1f1f38410bd","Type":"ContainerStarted","Data":"628153e9c7bd40d1735c99a48b322270297393fa436beba06023c088c07c9b4c"} Oct 30 05:30:29.433494 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.433473 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l5k87" event={"ID":"9e2abeba-b181-40af-89c2-c3782ad160c5","Type":"ContainerStarted","Data":"2c3e23841ad10bd56af282abb2e7ce1c8ba3a08cee26b872f69e2a511562b659"} Oct 30 05:30:29.433557 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.433522 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l5k87" event={"ID":"9e2abeba-b181-40af-89c2-c3782ad160c5","Type":"ContainerStarted","Data":"4aa810268db76f323f7ed1ed6135c0d43b257fe72cd6acc3d45d57412a328952"} Oct 30 05:30:29.434569 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.434551 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" event={"ID":"8d813b7e-2c08-4c0f-b077-2b529d90cf51","Type":"ContainerStarted","Data":"33f15deada5b62f8a545c185d95583be81f72bf5816b9ac2fc84ce6433ca2153"} Oct 30 05:30:29.435490 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.435453 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" event={"ID":"60d71bc3-a236-4305-baeb-df6403b52f43","Type":"ContainerStarted","Data":"3efc823edb824143ff5be061d45040dbb07f3f226cc5dfb01fc07c77e286852b"} Oct 30 05:30:29.436709 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.436692 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" event={"ID":"ae0a00ed-126f-4c35-a159-aa6ed7e2c72f","Type":"ContainerStarted","Data":"b00947966440ca514de2a6f3dbeed7661321518a3541f0508a861f81893ecfef"} Oct 30 05:30:29.436773 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.436719 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" event={"ID":"ae0a00ed-126f-4c35-a159-aa6ed7e2c72f","Type":"ContainerStarted","Data":"00fddd0a2156689a45838a017d5d90848112f2e7b6ff991cee5c446abeb22b98"} Oct 30 05:30:29.469592 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.469552 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" podStartSLOduration=809.469540111 podStartE2EDuration="13m29.469540111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:29.468820703 +0000 UTC m=+30.760373729" watchObservedRunningTime="2026-10-30 05:30:29.469540111 +0000 UTC m=+30.761093131" Oct 30 05:30:29.510364 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.510323 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" podStartSLOduration=809.510312801 podStartE2EDuration="13m29.510312801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:29.509606601 +0000 UTC m=+30.801159624" watchObservedRunningTime="2026-10-30 05:30:29.510312801 +0000 UTC m=+30.801865824" Oct 30 05:30:29.550946 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:29.550882 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-l5k87" podStartSLOduration=809.550872848 podStartE2EDuration="13m29.550872848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:29.550722036 +0000 UTC m=+30.842275057" watchObservedRunningTime="2026-10-30 05:30:29.550872848 +0000 UTC m=+30.842425872" Oct 30 05:30:30.210845 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.209599 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:30.210845 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.210203 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:30.210845 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.210576 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:30.226018 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.225997 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:30:30.226169 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.226047 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-897vl\"" Oct 30 05:30:30.226169 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.226126 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w4h88\"" Oct 30 05:30:30.226169 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.226156 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:30:30.226291 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.226170 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c4j6h\"" Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.292173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.292248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.292293 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.292325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.292377 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.292415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292604 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292666 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls podName:1e52c123-4401-4329-8058-5479f3680281 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.292647325 +0000 UTC m=+33.584200342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gx2pw" (UID: "1e52c123-4401-4329-8058-5479f3680281") : secret "samples-operator-tls" not found Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292727 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292756 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.292748399 +0000 UTC m=+33.584301404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : secret "router-metrics-certs-default" not found Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292814 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.292804551 +0000 UTC m=+33.584357569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : configmap references non-existent config key: service-ca.crt Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292854 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292874 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls podName:afe5578f-4885-4a55-a947-4c9bfe331614 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.292867908 +0000 UTC m=+33.584420911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4mn4d" (UID: "afe5578f-4885-4a55-a947-4c9bfe331614") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292920 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:30.293301 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292928 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc48cb9d8-jzp2t: secret "image-registry-tls" not found Oct 30 05:30:30.294408 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292947 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls podName:35863a55-5422-4304-ad71-afddab4d4807 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.292941737 +0000 UTC m=+33.584494740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls") pod "image-registry-cc48cb9d8-jzp2t" (UID: "35863a55-5422-4304-ad71-afddab4d4807") : secret "image-registry-tls" not found Oct 30 05:30:30.294408 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292984 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:30.294408 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.292993 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869865c857-td99r: secret "image-registry-tls" not found Oct 30 05:30:30.294408 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.293009 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls podName:2fd827f0-3153-4b94-8326-8c6fd0744f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.293003711 +0000 UTC m=+33.584556714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls") pod "image-registry-869865c857-td99r" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b") : secret "image-registry-tls" not found Oct 30 05:30:30.393986 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.393950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:30.394156 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:30.394043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:30.394529 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.394316 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:30:30.394529 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.394383 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls podName:0ee3f1bf-665e-4220-ac90-5d0b041d7077 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.394363283 +0000 UTC m=+33.685916300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls") pod "dns-default-bnfpg" (UID: "0ee3f1bf-665e-4220-ac90-5d0b041d7077") : secret "dns-default-metrics-tls" not found Oct 30 05:30:30.394529 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.394446 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:30:30.394529 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:30.394502 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert podName:429c555a-f735-446f-8577-4f7670a5d094 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.394490908 +0000 UTC m=+33.686043913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l725b" (UID: "429c555a-f735-446f-8577-4f7670a5d094") : secret "networking-console-plugin-cert" not found Oct 30 05:30:31.453389 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:31.450869 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" event={"ID":"d0f89396-3754-4034-af10-b9a951d3bbb1","Type":"ContainerStarted","Data":"0cf9b0ffb31cfbb6767eba858901f38fcb0c98badaa52e67edd86301806b7246"} Oct 30 05:30:32.312862 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:32.312820 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:32.313099 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:32.312951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:32.313099 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:32.312985 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:32.313099 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:32.313005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:32.313099 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:32.313040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:32.313099 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:32.313063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:32.313352 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313165 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:30:32.313352 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313213 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls podName:1e52c123-4401-4329-8058-5479f3680281 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.313200332 +0000 UTC m=+37.604753333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gx2pw" (UID: "1e52c123-4401-4329-8058-5479f3680281") : secret "samples-operator-tls" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313531 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313569 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.313560246 +0000 UTC m=+37.605113248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : secret "router-metrics-certs-default" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313609 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313618 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313631 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313642 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc48cb9d8-jzp2t: secret "image-registry-tls" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313652 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869865c857-td99r: secret "image-registry-tls" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313630 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls podName:afe5578f-4885-4a55-a947-4c9bfe331614 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.313624103 +0000 UTC m=+37.605177105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4mn4d" (UID: "afe5578f-4885-4a55-a947-4c9bfe331614") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313715 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls podName:2fd827f0-3153-4b94-8326-8c6fd0744f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.313698504 +0000 UTC m=+37.605251526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls") pod "image-registry-869865c857-td99r" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b") : secret "image-registry-tls" not found Oct 30 05:30:32.313730 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313738 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.313728794 +0000 UTC m=+37.605281803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : configmap references non-existent config key: service-ca.crt Oct 30 05:30:32.314296 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.313755 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls podName:35863a55-5422-4304-ad71-afddab4d4807 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.313746185 +0000 UTC m=+37.605299191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls") pod "image-registry-cc48cb9d8-jzp2t" (UID: "35863a55-5422-4304-ad71-afddab4d4807") : secret "image-registry-tls" not found Oct 30 05:30:32.414042 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:32.414012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:32.414221 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:32.414076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:32.414298 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.414227 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:30:32.414364 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.414298 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert podName:429c555a-f735-446f-8577-4f7670a5d094 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.414280655 +0000 UTC m=+37.705833673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l725b" (UID: "429c555a-f735-446f-8577-4f7670a5d094") : secret "networking-console-plugin-cert" not found Oct 30 05:30:32.414364 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.414363 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:30:32.414518 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:32.414386 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls podName:0ee3f1bf-665e-4220-ac90-5d0b041d7077 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.414379447 +0000 UTC m=+37.705932449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls") pod "dns-default-bnfpg" (UID: "0ee3f1bf-665e-4220-ac90-5d0b041d7077") : secret "dns-default-metrics-tls" not found Oct 30 05:30:33.022094 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.021917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:33.022094 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.021974 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:33.022643 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:33.022136 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 05:30:33.022643 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:33.022199 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs podName:671efddc-7c80-4be5-827d-0c227985c013 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:05.022179862 +0000 UTC m=+66.313732864 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs") pod "network-metrics-daemon-mtw44" (UID: "671efddc-7c80-4be5-827d-0c227985c013") : secret "metrics-daemon-secret" not found Oct 30 05:30:33.025065 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.025005 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:33.123217 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.123171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:33.126593 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.126562 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:33.237502 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.237312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:33.268395 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.268367 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:33.460517 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.460490 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/0.log" Oct 30 05:30:33.460670 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.460538 2581 generic.go:356] "Generic (PLEG): container finished" podID="8d813b7e-2c08-4c0f-b077-2b529d90cf51" containerID="653c310e669f39b294ddd7d7e2140c422a02a1b82c199abcc0e75b61f692f017" exitCode=255 Oct 30 05:30:33.460670 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.460575 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" event={"ID":"8d813b7e-2c08-4c0f-b077-2b529d90cf51","Type":"ContainerDied","Data":"653c310e669f39b294ddd7d7e2140c422a02a1b82c199abcc0e75b61f692f017"} Oct 30 05:30:33.460945 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.460926 2581 scope.go:122] "RemoveContainer" containerID="653c310e669f39b294ddd7d7e2140c422a02a1b82c199abcc0e75b61f692f017" Oct 30 05:30:33.504359 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.504307 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" podStartSLOduration=780.739700536 podStartE2EDuration="13m2.504288539s" totalImagesPullingTime="1.764588003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:31.49348584 +0000 UTC m=+32.785038862" watchObservedRunningTime="2026-10-30 05:30:33.504288539 +0000 UTC m=+34.795841567" Oct 30 05:30:33.577354 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.577322 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g6mhk"] Oct 30 05:30:33.639687 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:33.639660 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bmv92"] Oct 30 05:30:33.773581 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:33.773536 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d304ba5_b889_412c_ae7d_b26f04222714.slice/crio-400c440d0d11c896d69540aa9ab0e17271cdf46df7b56b123f20a6e5cda1067f WatchSource:0}: Error finding container 400c440d0d11c896d69540aa9ab0e17271cdf46df7b56b123f20a6e5cda1067f: Status 404 returned error can't find the container with id 400c440d0d11c896d69540aa9ab0e17271cdf46df7b56b123f20a6e5cda1067f Oct 30 05:30:33.773931 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:33.773902 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c8ed7c_08dd_4edd_8bfd_8a8b20b25cde.slice/crio-eaede63d735c2995d5b13d3db51a00c8c303ab805807f5e8d4063c13a6555bd9 WatchSource:0}: Error finding container eaede63d735c2995d5b13d3db51a00c8c303ab805807f5e8d4063c13a6555bd9: Status 404 returned error can't find the container with id eaede63d735c2995d5b13d3db51a00c8c303ab805807f5e8d4063c13a6555bd9 Oct 30 05:30:34.467532 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.467439 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g6mhk" event={"ID":"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde","Type":"ContainerStarted","Data":"eaede63d735c2995d5b13d3db51a00c8c303ab805807f5e8d4063c13a6555bd9"} Oct 30 05:30:34.469364 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.469289 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:30:34.471586 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.470101 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/0.log" Oct 30 05:30:34.471586 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.470138 2581 generic.go:356] "Generic (PLEG): container finished" podID="8d813b7e-2c08-4c0f-b077-2b529d90cf51" containerID="76737ca3daa3bc1b7826bbb28c96768c103e4eb44fcfedc551d1ee20fc521585" exitCode=255 Oct 30 05:30:34.471586 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.470544 2581 scope.go:122] "RemoveContainer" containerID="76737ca3daa3bc1b7826bbb28c96768c103e4eb44fcfedc551d1ee20fc521585" Oct 30 05:30:34.471586 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:34.470810 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c9p4j_openshift-console-operator(8d813b7e-2c08-4c0f-b077-2b529d90cf51)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" podUID="8d813b7e-2c08-4c0f-b077-2b529d90cf51" Oct 30 05:30:34.471586 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.470863 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" event={"ID":"8d813b7e-2c08-4c0f-b077-2b529d90cf51","Type":"ContainerDied","Data":"76737ca3daa3bc1b7826bbb28c96768c103e4eb44fcfedc551d1ee20fc521585"} Oct 30 05:30:34.471586 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.470909 2581 scope.go:122] "RemoveContainer" containerID="653c310e669f39b294ddd7d7e2140c422a02a1b82c199abcc0e75b61f692f017" Oct 30 05:30:34.477265 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.477096 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" event={"ID":"60d71bc3-a236-4305-baeb-df6403b52f43","Type":"ContainerStarted","Data":"f6bcda01cf71459069982c449fa9598488ff34e98074ca971842a59dba058767"} Oct 30 05:30:34.479403 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.479376 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bmv92" event={"ID":"0d304ba5-b889-412c-ae7d-b26f04222714","Type":"ContainerStarted","Data":"400c440d0d11c896d69540aa9ab0e17271cdf46df7b56b123f20a6e5cda1067f"} Oct 30 05:30:34.481651 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.481623 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" event={"ID":"2f968039-9ce3-48ea-a1c6-c963206a596c","Type":"ContainerStarted","Data":"17456246d697630c019eebd1ef4c80b2629553209650a1be3da7486990e05e80"} Oct 30 05:30:34.489764 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.489730 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" event={"ID":"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d","Type":"ContainerStarted","Data":"0de7de9f69cb278be6f620c4043379c0f8b3efc6654339e4cf6d4a7a9c077931"} Oct 30 05:30:34.557608 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.557553 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" podStartSLOduration=781.680141685 podStartE2EDuration="13m6.55753445s" totalImagesPullingTime="4.877392765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:34.556144513 +0000 UTC m=+35.847697550" watchObservedRunningTime="2026-10-30 05:30:34.55753445 +0000 UTC m=+35.849087477" Oct 30 05:30:34.599917 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.599835 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" podStartSLOduration=776.768268402 podStartE2EDuration="13m1.59981646s" totalImagesPullingTime="4.831548058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:34.598336364 +0000 UTC m=+35.889889390" watchObservedRunningTime="2026-10-30 05:30:34.59981646 +0000 UTC m=+35.891369483" Oct 30 05:30:34.682386 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:34.682340 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" podStartSLOduration=781.808835457 podStartE2EDuration="13m6.682324762s" totalImagesPullingTime="4.873489305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:34.680524214 +0000 UTC m=+35.972077428" watchObservedRunningTime="2026-10-30 05:30:34.682324762 +0000 UTC m=+35.973877785" Oct 30 05:30:35.240598 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.240517 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tlm2v"] Oct 30 05:30:35.262169 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.262137 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt"] Oct 30 05:30:35.262678 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.262657 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.282646 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.281668 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv"] Oct 30 05:30:35.282646 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.282046 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" Oct 30 05:30:35.298050 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.297987 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt"] Oct 30 05:30:35.298050 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.298031 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tlm2v"] Oct 30 05:30:35.298247 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.298136 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" Oct 30 05:30:35.315075 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.315014 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnf9b\"" Oct 30 05:30:35.315075 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.315032 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cjds4\"" Oct 30 05:30:35.315075 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.315039 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:30:35.330735 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.328447 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv"] Oct 30 05:30:35.331814 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.331789 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-6hlgb-qtlr7"] Oct 30 05:30:35.339362 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.339321 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:30:35.339362 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.339329 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:35.339543 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.339418 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 05:30:35.348299 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.348088 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.356372 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.356306 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6hlgb-qtlr7"] Oct 30 05:30:35.446385 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b26dcb-4282-498a-b278-4c7bb6ac11a1-data-volume\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.446385 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446392 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7gk\" (UniqueName: \"kubernetes.io/projected/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-api-access-pt7gk\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.446631 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446526 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.446631 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.446701 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhq8c\" (UniqueName: \"kubernetes.io/projected/af48072f-2e02-46db-9b2d-7eb2da4cd2e4-kube-api-access-nhq8c\") pod \"migrator-66df5d496b-5z7dt\" (UID: \"af48072f-2e02-46db-9b2d-7eb2da4cd2e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" Oct 30 05:30:35.446701 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.446797 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kstz\" (UniqueName: \"kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.446797 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.446893 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446794 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjbzf\" (UniqueName: \"kubernetes.io/projected/454eb54a-074f-45e9-94cb-75356bb42e71-kube-api-access-gjbzf\") pod \"migrator-66df5d496b-kl6gv\" (UID: \"454eb54a-074f-45e9-94cb-75356bb42e71\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" Oct 30 05:30:35.446893 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446843 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.446893 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.446873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95b26dcb-4282-498a-b278-4c7bb6ac11a1-crio-socket\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.499528 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.498950 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:30:35.499528 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.499315 2581 scope.go:122] "RemoveContainer" containerID="76737ca3daa3bc1b7826bbb28c96768c103e4eb44fcfedc551d1ee20fc521585" Oct 30 05:30:35.500008 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:35.499569 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c9p4j_openshift-console-operator(8d813b7e-2c08-4c0f-b077-2b529d90cf51)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" podUID="8d813b7e-2c08-4c0f-b077-2b529d90cf51" Oct 30 05:30:35.501943 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.501446 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x2s8x" event={"ID":"540c21f9-43da-400c-b0da-e0b01eecedf7","Type":"ContainerStarted","Data":"568e7266112abc012ae5d294acfd827dd9d7081d0f2123ba13be84a7b023e93f"} Oct 30 05:30:35.504074 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.504031 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" event={"ID":"26ef5d7f-4915-4cee-a412-c1f1f38410bd","Type":"ContainerStarted","Data":"0dfed1338f06bb577603b910caf44c1498bede8462cda7c6b3820f7db8845e8b"} Oct 30 05:30:35.548623 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.548587 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b26dcb-4282-498a-b278-4c7bb6ac11a1-data-volume\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.548779 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.548633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7gk\" (UniqueName: \"kubernetes.io/projected/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-api-access-pt7gk\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.548779 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.548684 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.548779 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.548748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.548964 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:35.548829 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:30:35.548964 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:35.548905 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.048887177 +0000 UTC m=+37.340440194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : configmap "kube-rbac-proxy" not found Oct 30 05:30:35.549564 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:35.549539 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:30:35.549691 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.549647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhq8c\" (UniqueName: \"kubernetes.io/projected/af48072f-2e02-46db-9b2d-7eb2da4cd2e4-kube-api-access-nhq8c\") pod \"migrator-66df5d496b-5z7dt\" (UID: \"af48072f-2e02-46db-9b2d-7eb2da4cd2e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" Oct 30 05:30:35.549691 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.549677 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.549843 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.549704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kstz\" (UniqueName: \"kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.549843 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.549754 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.550363 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.550339 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.550651 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.549422 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b26dcb-4282-498a-b278-4c7bb6ac11a1-data-volume\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.550715 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:35.550683 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:36.050664323 +0000 UTC m=+37.342217354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : secret "insights-runtime-extractor-tls" not found Oct 30 05:30:35.550830 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.550809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjbzf\" (UniqueName: \"kubernetes.io/projected/454eb54a-074f-45e9-94cb-75356bb42e71-kube-api-access-gjbzf\") pod \"migrator-66df5d496b-kl6gv\" (UID: \"454eb54a-074f-45e9-94cb-75356bb42e71\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" Oct 30 05:30:35.553686 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.553647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.553686 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.553680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95b26dcb-4282-498a-b278-4c7bb6ac11a1-crio-socket\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.553874 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.553752 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95b26dcb-4282-498a-b278-4c7bb6ac11a1-crio-socket\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.554107 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.554084 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.554427 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.554404 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.580485 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.580433 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjbzf\" (UniqueName: \"kubernetes.io/projected/454eb54a-074f-45e9-94cb-75356bb42e71-kube-api-access-gjbzf\") pod \"migrator-66df5d496b-kl6gv\" (UID: \"454eb54a-074f-45e9-94cb-75356bb42e71\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" Oct 30 05:30:35.582857 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.582829 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhq8c\" (UniqueName: \"kubernetes.io/projected/af48072f-2e02-46db-9b2d-7eb2da4cd2e4-kube-api-access-nhq8c\") pod \"migrator-66df5d496b-5z7dt\" (UID: \"af48072f-2e02-46db-9b2d-7eb2da4cd2e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" Oct 30 05:30:35.585284 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.585260 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7gk\" (UniqueName: \"kubernetes.io/projected/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-api-access-pt7gk\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:35.586858 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.586833 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kstz\" (UniqueName: \"kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.631973 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.631900 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" podStartSLOduration=774.997826892 podStartE2EDuration="13m0.631881789s" totalImagesPullingTime="5.634054897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:35.631634683 +0000 UTC m=+36.923187710" watchObservedRunningTime="2026-10-30 05:30:35.631881789 +0000 UTC m=+36.923434814" Oct 30 05:30:35.632650 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.632597 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x2s8x" podStartSLOduration=2.365101615 podStartE2EDuration="7.632585194s" totalImagesPullingTime="5.267483579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:35.585611124 +0000 UTC m=+36.877164160" watchObservedRunningTime="2026-10-30 05:30:35.632585194 +0000 UTC m=+36.924138219" Oct 30 05:30:35.658881 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.658673 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" Oct 30 05:30:35.678294 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.678257 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" Oct 30 05:30:35.700358 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.700333 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:35.731357 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.731331 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wgrjc"] Oct 30 05:30:35.751809 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.751571 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:35.767061 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.767032 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wgrjc"] Oct 30 05:30:35.767600 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.767581 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 05:30:35.783228 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.783188 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t86pd\"" Oct 30 05:30:35.795920 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.795892 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 05:30:35.796088 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.796069 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 05:30:35.796150 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.796140 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 05:30:35.858372 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.858044 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqgp\" (UniqueName: \"kubernetes.io/projected/70464c99-1f1b-49e4-8cc9-335fa1645a98-kube-api-access-rdqgp\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:35.858372 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.858110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-key\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:35.858372 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.858197 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:35.959322 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.959292 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqgp\" (UniqueName: \"kubernetes.io/projected/70464c99-1f1b-49e4-8cc9-335fa1645a98-kube-api-access-rdqgp\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:35.959929 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.959338 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-key\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:35.959929 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.959416 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:35.960196 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.960170 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:35.962700 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.962675 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-key\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:35.984873 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:35.984840 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqgp\" (UniqueName: \"kubernetes.io/projected/70464c99-1f1b-49e4-8cc9-335fa1645a98-kube-api-access-rdqgp\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:36.008892 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.008864 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv"] Oct 30 05:30:36.009394 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:36.009356 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454eb54a_074f_45e9_94cb_75356bb42e71.slice/crio-3363a6b3aad6626ee9f69ab64495d1f09bfb6953a34c5ca365d4132e217f3700 WatchSource:0}: Error finding container 3363a6b3aad6626ee9f69ab64495d1f09bfb6953a34c5ca365d4132e217f3700: Status 404 returned error can't find the container with id 3363a6b3aad6626ee9f69ab64495d1f09bfb6953a34c5ca365d4132e217f3700 Oct 30 05:30:36.044433 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.044409 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt"] Oct 30 05:30:36.060856 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.060830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:36.060976 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.060884 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:36.061037 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.061005 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:30:36.061069 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.061044 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:30:36.061101 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.061081 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:37.061055937 +0000 UTC m=+38.352608952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : secret "insights-runtime-extractor-tls" not found Oct 30 05:30:36.061101 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.061100 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:37.061088105 +0000 UTC m=+38.352641110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : configmap "kube-rbac-proxy" not found Oct 30 05:30:36.097450 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.097421 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6hlgb-qtlr7"] Oct 30 05:30:36.098707 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:36.098672 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d914f2_a350_49db_b937_c4774be4415e.slice/crio-4c76d6cc77a310ed1edf9da56a5c4a3742d5f51753860eabe60dc31345899e48 WatchSource:0}: Error finding container 4c76d6cc77a310ed1edf9da56a5c4a3742d5f51753860eabe60dc31345899e48: Status 404 returned error can't find the container with id 4c76d6cc77a310ed1edf9da56a5c4a3742d5f51753860eabe60dc31345899e48 Oct 30 05:30:36.122179 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.122156 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:36.365350 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.365298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:36.365595 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.365368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:36.365595 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365471 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:36.365595 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365493 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869865c857-td99r: secret "image-registry-tls" not found Oct 30 05:30:36.365595 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.365491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:36.365595 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365518 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:30:36.365595 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.365545 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:36.365595 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365564 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls podName:2fd827f0-3153-4b94-8326-8c6fd0744f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:44.365545284 +0000 UTC m=+45.657098301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls") pod "image-registry-869865c857-td99r" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b") : secret "image-registry-tls" not found Oct 30 05:30:36.365902 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365639 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls podName:1e52c123-4401-4329-8058-5479f3680281 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:44.365621893 +0000 UTC m=+45.657174897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gx2pw" (UID: "1e52c123-4401-4329-8058-5479f3680281") : secret "samples-operator-tls" not found Oct 30 05:30:36.365902 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365644 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 05:30:36.365902 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365659 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:44.365649516 +0000 UTC m=+45.657202520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : configmap references non-existent config key: service-ca.crt Oct 30 05:30:36.365902 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365715 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs podName:6381b6d8-a45f-43d3-955c-504694db87da nodeName:}" failed. No retries permitted until 2026-10-30 05:30:44.365684866 +0000 UTC m=+45.657237882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs") pod "router-default-54dbcc76d9-x7c5h" (UID: "6381b6d8-a45f-43d3-955c-504694db87da") : secret "router-metrics-certs-default" not found Oct 30 05:30:36.365902 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.365772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:36.365902 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.365805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:36.366096 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365925 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:36.366096 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365952 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:36.366096 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365965 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc48cb9d8-jzp2t: secret "image-registry-tls" not found Oct 30 05:30:36.366096 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.365999 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls podName:afe5578f-4885-4a55-a947-4c9bfe331614 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:44.365987533 +0000 UTC m=+45.657540548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4mn4d" (UID: "afe5578f-4885-4a55-a947-4c9bfe331614") : secret "cluster-monitoring-operator-tls" not found Oct 30 05:30:36.366096 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.366017 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls podName:35863a55-5422-4304-ad71-afddab4d4807 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:44.366008499 +0000 UTC m=+45.657561502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls") pod "image-registry-cc48cb9d8-jzp2t" (UID: "35863a55-5422-4304-ad71-afddab4d4807") : secret "image-registry-tls" not found Oct 30 05:30:36.462130 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.462100 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wgrjc"] Oct 30 05:30:36.463256 ip-10-0-138-200 kubenswrapper[2581]: W1030 05:30:36.463224 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70464c99_1f1b_49e4_8cc9_335fa1645a98.slice/crio-376c41e33d35a5249bc47fac684bcc0e50fa4b4b7c2fc89cbb1ddfdbd6791608 WatchSource:0}: Error finding container 376c41e33d35a5249bc47fac684bcc0e50fa4b4b7c2fc89cbb1ddfdbd6791608: Status 404 returned error can't find the container with id 376c41e33d35a5249bc47fac684bcc0e50fa4b4b7c2fc89cbb1ddfdbd6791608 Oct 30 05:30:36.466493 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.466450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:36.466612 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.466517 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:36.466669 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.466648 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 05:30:36.466728 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.466695 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert podName:429c555a-f735-446f-8577-4f7670a5d094 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:44.466683086 +0000 UTC m=+45.758236088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-l725b" (UID: "429c555a-f735-446f-8577-4f7670a5d094") : secret "networking-console-plugin-cert" not found Oct 30 05:30:36.466837 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.466814 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:30:36.466939 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:36.466888 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls podName:0ee3f1bf-665e-4220-ac90-5d0b041d7077 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:44.466874921 +0000 UTC m=+45.758427927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls") pod "dns-default-bnfpg" (UID: "0ee3f1bf-665e-4220-ac90-5d0b041d7077") : secret "dns-default-metrics-tls" not found Oct 30 05:30:36.513152 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.513120 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" event={"ID":"b9d914f2-a350-49db-b937-c4774be4415e","Type":"ContainerStarted","Data":"fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d"} Oct 30 05:30:36.513589 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.513178 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" event={"ID":"b9d914f2-a350-49db-b937-c4774be4415e","Type":"ContainerStarted","Data":"4c76d6cc77a310ed1edf9da56a5c4a3742d5f51753860eabe60dc31345899e48"} Oct 30 05:30:36.514595 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.514570 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" event={"ID":"454eb54a-074f-45e9-94cb-75356bb42e71","Type":"ContainerStarted","Data":"3363a6b3aad6626ee9f69ab64495d1f09bfb6953a34c5ca365d4132e217f3700"} Oct 30 05:30:36.516256 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.515870 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" event={"ID":"af48072f-2e02-46db-9b2d-7eb2da4cd2e4","Type":"ContainerStarted","Data":"46d6af6624ec74c3ae1ec215aaaf1c3b9964a093813c1d468a8963329d139b68"} Oct 30 05:30:36.517358 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.517331 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" event={"ID":"70464c99-1f1b-49e4-8cc9-335fa1645a98","Type":"ContainerStarted","Data":"376c41e33d35a5249bc47fac684bcc0e50fa4b4b7c2fc89cbb1ddfdbd6791608"} Oct 30 05:30:36.558558 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:36.557760 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" podStartSLOduration=1.5577448710000001 podStartE2EDuration="1.557744871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:36.555320356 +0000 UTC m=+37.846873393" watchObservedRunningTime="2026-10-30 05:30:36.557744871 +0000 UTC m=+37.849297908" Oct 30 05:30:37.075533 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:37.075497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:37.075725 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:37.075710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:37.075871 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:37.075853 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:30:37.075943 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:37.075932 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:39.0759119 +0000 UTC m=+40.367464917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : secret "insights-runtime-extractor-tls" not found Oct 30 05:30:37.076271 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:37.076255 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:30:37.076338 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:37.076298 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:39.076288854 +0000 UTC m=+40.367841869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : configmap "kube-rbac-proxy" not found Oct 30 05:30:37.531177 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:37.530593 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" event={"ID":"70464c99-1f1b-49e4-8cc9-335fa1645a98","Type":"ContainerStarted","Data":"65ab6e7220c344cb19ff1cb5b22a61448c8ed155f2bd7e4ef2f02dfb47ece218"} Oct 30 05:30:37.533613 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:37.533583 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bmv92" event={"ID":"0d304ba5-b889-412c-ae7d-b26f04222714","Type":"ContainerStarted","Data":"c36f920f87a31c3f95d39ec2d5ccf23461157feebc412053bf98261382cb0abe"} Oct 30 05:30:37.533711 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:37.533636 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:37.584801 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:37.584733 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" podStartSLOduration=2.584716866 podStartE2EDuration="2.584716866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:37.584283502 +0000 UTC m=+38.875836526" watchObservedRunningTime="2026-10-30 05:30:37.584716866 +0000 UTC m=+38.876269892" Oct 30 05:30:37.666574 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:37.666516 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bmv92" podStartSLOduration=35.722580349 podStartE2EDuration="38.666501404s" totalImagesPullingTime="2.943921055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:37.665938817 +0000 UTC m=+38.957491842" watchObservedRunningTime="2026-10-30 05:30:37.666501404 +0000 UTC m=+38.958054425" Oct 30 05:30:38.540602 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:38.540563 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" event={"ID":"454eb54a-074f-45e9-94cb-75356bb42e71","Type":"ContainerStarted","Data":"0eb713e03da3e9f474a5579f03e22b2d552601ef8bd17b337c45a1995c139e31"} Oct 30 05:30:38.542392 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:38.542361 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" event={"ID":"af48072f-2e02-46db-9b2d-7eb2da4cd2e4","Type":"ContainerStarted","Data":"598b886ee236169325f03ee73825c9fb080b52a4161b00df3264d0a4fa6a2adb"} Oct 30 05:30:38.582938 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:38.582889 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" podStartSLOduration=1.855218719 podStartE2EDuration="3.582870647s" totalImagesPullingTime="1.727651928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:38.580939964 +0000 UTC m=+39.872492990" watchObservedRunningTime="2026-10-30 05:30:38.582870647 +0000 UTC m=+39.874423675" Oct 30 05:30:38.621087 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:38.621044 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" podStartSLOduration=1.770106296 podStartE2EDuration="3.621030679s" totalImagesPullingTime="1.850924383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:38.620718441 +0000 UTC m=+39.912271469" watchObservedRunningTime="2026-10-30 05:30:38.621030679 +0000 UTC m=+39.912583693" Oct 30 05:30:38.939174 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:38.939146 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:38.939338 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:38.939203 2581 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:38.939612 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:38.939598 2581 scope.go:122] "RemoveContainer" containerID="76737ca3daa3bc1b7826bbb28c96768c103e4eb44fcfedc551d1ee20fc521585" Oct 30 05:30:38.939845 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:38.939811 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c9p4j_openshift-console-operator(8d813b7e-2c08-4c0f-b077-2b529d90cf51)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" podUID="8d813b7e-2c08-4c0f-b077-2b529d90cf51" Oct 30 05:30:39.097232 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:39.097197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:39.097415 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:39.097265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:39.097415 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:39.097353 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:30:39.097415 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:39.097394 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 05:30:39.097602 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:39.097469 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:43.09743691 +0000 UTC m=+44.388989932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : configmap "kube-rbac-proxy" not found Oct 30 05:30:39.097602 ip-10-0-138-200 kubenswrapper[2581]: E1030 05:30:39.097487 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:43.097480203 +0000 UTC m=+44.389033205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : secret "insights-runtime-extractor-tls" not found Oct 30 05:30:40.295608 ip-10-0-138-200 kubenswrapper[2581]: I1030 05:30:40.295419 2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:30:40.295620 ip-10-0-138-200 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 05:30:40.317857 ip-10-0-138-200 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 05:30:40.318032 ip-10-0-138-200 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 05:30:40.318080 ip-10-0-138-200 systemd[1]: kubelet.service: Consumed 3.264s CPU time. Oct 30 05:30:40.336079 ip-10-0-138-200 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:30:40.425836 ip-10-0-138-200 kubenswrapper[5072]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:30:40.425836 ip-10-0-138-200 kubenswrapper[5072]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:30:40.425836 ip-10-0-138-200 kubenswrapper[5072]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:30:40.425836 ip-10-0-138-200 kubenswrapper[5072]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:30:40.427835 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427758 5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:40.427835 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427836 5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427840 5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427844 5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427847 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427850 5072 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427853 5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427859 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427862 5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427865 5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427868 5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427870 5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427873 5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427875 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427878 5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427880 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427883 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427885 5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427888 5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427890 5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427893 5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:40.427909 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427895 5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427898 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427901 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427903 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427906 5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427908 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427911 5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427913 5072 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427916 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427919 5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427922 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427925 5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427927 5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427930 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427932 5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427935 5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427937 5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427939 5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427942 5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:40.428390 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427945 5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427947 5072 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427950 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427952 5072 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427955 5072 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427957 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427960 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427962 5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427964 5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427967 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427969 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427972 5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427975 5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427979 5072 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427983 5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427987 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427991 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427994 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427997 5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.427999 5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:40.428869 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428002 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428005 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428007 5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428010 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428012 5072 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428015 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428018 5072 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428020 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428023 5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428025 5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428028 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428031 5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428033 5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428036 5072 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428039 5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428041 5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428044 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428046 5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428049 5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428051 5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:40.429365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428053 5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428056 5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428058 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428062 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428064 5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428067 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428070 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428073 5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428075 5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428078 5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428080 5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428083 5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428088 5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428091 5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428093 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428096 5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428098 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428101 5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428103 5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:40.429865 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428106 5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428110 5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428115 5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428117 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428120 5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428123 5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428126 5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428896 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428902 5072 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428905 5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428908 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428911 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428913 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428916 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428919 5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428921 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428924 5072 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428926 5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428929 5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428932 5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:40.430377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428935 5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428938 5072 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428940 5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428943 5072 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428946 5072 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428951 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428953 5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428956 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428958 5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428961 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428963 5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428966 5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428969 5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428971 5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428974 5072 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428976 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428979 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428981 5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428983 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428987 5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:40.430885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428989 5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428992 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428994 5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428997 5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.428999 5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429002 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429004 5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429007 5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429009 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429012 5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429014 5072 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429017 5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429019 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429022 5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429026 5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429030 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429033 5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429037 5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429040 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:40.431365 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429042 5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429045 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429047 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429050 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429052 5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429055 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429057 5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429061 5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429065 5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429068 5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429070 5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429073 5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429075 5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429079 5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429081 5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429083 5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429086 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429089 5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429092 5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:40.431862 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429094 5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429097 5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429100 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429102 5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429104 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429107 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429110 5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429112 5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429114 5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429117 5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429119 5072 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429124 5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429127 5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429129 5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429132 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429134 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429137 5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429139 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429142 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:40.432350 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429144 5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429146 5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429149 5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429151 5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429154 5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429157 5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429159 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429162 5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429165 5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429168 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429170 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429173 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429175 5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429178 5072 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429181 5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429183 5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429695 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429701 5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429704 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429707 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:40.432877 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429710 5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429712 5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429715 5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429718 5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429722 5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429725 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429728 5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429731 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429733 5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429736 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429739 5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429741 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429744 5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429747 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429749 5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429752 5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429754 5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429757 5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429759 5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429762 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:40.433362 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429765 5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429768 5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429770 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429773 5072 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429775 5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429778 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429780 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429782 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429785 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429788 5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429791 5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429794 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429796 5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429799 5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429801 5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429804 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429808 5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429810 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429813 5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429815 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:40.433955 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429818 5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429820 5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429823 5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429825 5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429828 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429830 5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429833 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429835 5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429838 5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429840 5072 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429843 5072 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429845 5072 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429847 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429850 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429854 5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429857 5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429859 5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429862 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429864 5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429867 5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:40.434801 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429869 5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429872 5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429876 5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429878 5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429881 5072 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429883 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429886 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429888 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429892 5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429894 5072 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429897 5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429900 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429902 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429905 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429907 5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429909 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429912 5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429914 5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429917 5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:40.435505 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429919 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429922 5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429924 5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429927 5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429929 5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429932 5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429934 5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429937 5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429939 5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429942 5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429944 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429946 5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429949 5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429952 5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429956 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429960 5072 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429965 5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429968 5072 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429971 5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:40.436084 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429974 5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429977 5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429981 5072 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.429984 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430077 5072 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430089 5072 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430096 5072 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430100 5072 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430105 5072 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430108 5072 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430112 5072 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430116 5072 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430119 5072 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430122 5072 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430125 5072 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430129 5072 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430132 5072 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430135 5072 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430138 5072 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430141 5072 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430143 5072 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430147 5072 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430150 5072 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430153 5072 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:30:40.436580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430156 5072 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430159 5072 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430163 5072 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430167 5072 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430171 5072 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430174 5072 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430178 5072 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430181 5072 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430184 5072 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430186 5072 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430192 5072 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430195 5072 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430199 5072 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430202 5072 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430205 5072 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430208 5072 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430211 5072 flags.go:64] FLAG: --enable-server="true" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430214 5072 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430218 5072 flags.go:64] FLAG: --event-burst="100" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430222 5072 flags.go:64] FLAG: --event-qps="50" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430225 5072 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430228 5072 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430230 5072 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430235 5072 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:30:40.437143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430239 5072 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430242 5072 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430244 5072 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430247 5072 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430250 5072 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430254 5072 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430257 5072 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430260 5072 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430263 5072 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430266 5072 flags.go:64] FLAG: --feature-gates="" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430270 5072 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430273 5072 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430277 5072 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430281 5072 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430284 5072 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430287 5072 flags.go:64] FLAG: --help="false" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430290 5072 flags.go:64] FLAG: --hostname-override="ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430293 5072 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430297 5072 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430300 5072 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430303 5072 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430306 5072 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430309 5072 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:30:40.437739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430312 5072 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430315 5072 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430318 5072 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430320 5072 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430323 5072 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430326 5072 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430329 5072 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430332 5072 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430334 5072 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430337 5072 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430340 5072 flags.go:64] FLAG: --lock-file="" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430343 5072 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430346 5072 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430349 5072 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430354 5072 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430357 5072 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430360 5072 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430363 5072 flags.go:64] FLAG: --logging-format="text" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430366 5072 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430369 5072 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430372 5072 flags.go:64] FLAG: --manifest-url="" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430377 5072 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430382 5072 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430386 5072 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430389 5072 flags.go:64] FLAG: --max-pods="110" Oct 30 05:30:40.438327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430392 5072 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430395 5072 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430399 5072 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430402 5072 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430405 5072 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430408 5072 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430411 5072 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430418 5072 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430421 5072 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430424 5072 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430427 5072 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430430 5072 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430432 5072 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430435 5072 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430438 5072 flags.go:64] FLAG: --port="10250" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430441 5072 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430444 5072 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08573b397eedecbd1" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430447 5072 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430450 5072 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430467 5072 flags.go:64] FLAG: --register-node="true" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430470 5072 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430474 5072 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430476 5072 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430479 5072 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430483 5072 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430487 5072 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:30:40.438969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430490 5072 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430493 5072 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430497 5072 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430500 5072 flags.go:64] FLAG: --runonce="false" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430503 5072 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430506 5072 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430509 5072 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430512 5072 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430517 5072 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430520 5072 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430523 5072 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430526 5072 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430529 5072 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430532 5072 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430535 5072 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430538 5072 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430541 5072 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430544 5072 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430547 5072 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430552 5072 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430555 5072 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430558 5072 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430561 5072 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430564 5072 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430567 5072 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:30:40.439701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430570 5072 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430573 5072 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430576 5072 flags.go:64] FLAG: --v="2" Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430580 5072 flags.go:64] FLAG: --version="false" Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430584 5072 flags.go:64] FLAG: --vmodule="" Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430588 5072 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.430592 5072 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430722 5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430727 5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430731 5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430734 5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430737 5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430740 5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430742 5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430744 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430748 5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430751 5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430753 5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430756 5072 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430758 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430761 5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:40.440305 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430763 5072 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430766 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430768 5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430771 5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430773 5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430776 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430778 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430781 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430783 5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430785 5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430788 5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430790 5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430793 5072 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430795 5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430797 5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430800 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430804 5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430807 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430811 5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:40.440874 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430814 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430818 5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430820 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430823 5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430825 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430828 5072 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430830 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430834 5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430837 5072 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430839 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430842 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430844 5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430847 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430850 5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430852 5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430855 5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430857 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430860 5072 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430862 5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430864 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:40.441523 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430867 5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430869 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430872 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430874 5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430877 5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430879 5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430881 5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430884 5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430886 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430889 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430892 5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430894 5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430898 5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430902 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430904 5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430907 5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430909 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430912 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430914 5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:40.442087 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430917 5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430919 5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430922 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430924 5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430927 5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430929 5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430932 5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430934 5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430936 5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430939 5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430941 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430944 5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430946 5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430948 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430951 5072 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430953 5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430956 5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430958 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430960 5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430963 5072 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:40.442621 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430965 5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430968 5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430970 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430972 5072 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430975 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430977 5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430981 5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430984 5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430986 5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430989 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430991 5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430993 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430996 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.430998 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.431024 5072 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431079 5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:40.443176 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431083 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431087 5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431089 5072 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431092 5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431094 5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431097 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431099 5072 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431102 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431105 5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431107 5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431110 5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431112 5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431115 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431117 5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431119 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431122 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431124 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431127 5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431130 5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:40.443643 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431134 5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431137 5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431140 5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431144 5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431147 5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431150 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431152 5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431155 5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431157 5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431160 5072 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431162 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431165 5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431168 5072 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431170 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431172 5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431175 5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431177 5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431180 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431182 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431185 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:40.444156 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431187 5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431190 5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431192 5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431194 5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431197 5072 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431199 5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431202 5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431204 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431209 5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431213 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431216 5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431219 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431221 5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431224 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431226 5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431230 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431232 5072 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431235 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431238 5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431240 5072 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:40.444774 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431243 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431245 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431248 5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431250 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431253 5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431255 5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431258 5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431262 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431264 5072 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431267 5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431269 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431271 5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431274 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431277 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431279 5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431281 5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431284 5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431286 5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431289 5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:40.445322 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431291 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431293 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431296 5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431298 5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431301 5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431303 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431306 5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431308 5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431312 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431315 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431317 5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431320 5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431323 5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431325 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431328 5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431330 5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431333 5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431335 5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431338 5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:40.445922 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431340 5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431344 5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431347 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431349 5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431352 5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431354 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431357 5072 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.431360 5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.436639 5072 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.436658 5072 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.436678 5072 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.436684 5072 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436746 5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436751 5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436755 5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436759 5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436764 5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436767 5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436770 5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:40.446447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436773 5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436776 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436778 5072 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436781 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436784 5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436787 5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436789 5072 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436792 5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436795 5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436798 5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436801 5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436804 5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436806 5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436809 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436811 5072 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436814 5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436817 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436819 5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436822 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436824 5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:40.446967 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436827 5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436829 5072 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436831 5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436834 5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436837 5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436839 5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436841 5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436844 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436846 5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436849 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436851 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436854 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436856 5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436859 5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436861 5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436865 5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436867 5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436870 5072 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436873 5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436876 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:40.447447 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436879 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436881 5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436884 5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436886 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436889 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436891 5072 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436894 5072 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436896 5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436899 5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436901 5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436904 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436906 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436909 5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436911 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436914 5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436916 5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436919 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436922 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436924 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:40.447945 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436927 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436929 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436931 5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436934 5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436938 5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436942 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436944 5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436947 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436950 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436953 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436955 5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436958 5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436960 5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436964 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436967 5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436970 5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436973 5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436975 5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436977 5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:40.448412 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436980 5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436982 5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436985 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436987 5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436990 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436992 5072 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436995 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436997 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.436999 5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437002 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437004 5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437007 5072 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437009 5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437012 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437014 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437017 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437019 5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437022 5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437024 5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437026 5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:40.448894 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437029 5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.437060 5072 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437222 5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437228 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437231 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437233 5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437236 5072 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437239 5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437242 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437244 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437247 5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437250 5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437252 5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437255 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437257 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437260 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:40.449368 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437262 5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437265 5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437267 5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437269 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437272 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437274 5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437277 5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437280 5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437282 5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437284 5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437287 5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437289 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437292 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437294 5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437297 5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437300 5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437305 5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437307 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437310 5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:40.449845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437313 5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437315 5072 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437317 5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437320 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437323 5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437328 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437332 5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437336 5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437340 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437343 5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437347 5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437351 5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437355 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437360 5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437366 5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437372 5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437375 5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437378 5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437381 5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:40.450373 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437383 5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437386 5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437388 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437391 5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437393 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437396 5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437398 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437401 5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437403 5072 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437406 5072 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437408 5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437411 5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437413 5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437416 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437418 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437420 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437423 5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437426 5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437428 5072 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437432 5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:40.450876 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437435 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437438 5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437440 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437443 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437445 5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437448 5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437450 5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437452 5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437470 5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437472 5072 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437475 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437477 5072 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437480 5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437482 5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437485 5072 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437488 5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437490 5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437493 5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437495 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437498 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:40.451363 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437500 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437503 5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437506 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437508 5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437511 5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437514 5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437516 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437518 5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437521 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437526 5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437528 5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437531 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437534 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437537 5072 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.437562 5072 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:30:40.451897 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437598 5072 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437602 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437605 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437607 5072 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437610 5072 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437613 5072 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437615 5072 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437618 5072 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437621 5072 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437623 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437626 5072 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437628 5072 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437631 5072 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437633 5072 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437636 5072 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437639 5072 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437641 5072 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437644 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437646 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437650 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:40.452565 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437653 5072 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437655 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437658 5072 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437660 5072 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437663 5072 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437665 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437667 5072 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437671 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437674 5072 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437676 5072 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437680 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437682 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437685 5072 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437687 5072 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437690 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437692 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437694 5072 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437697 5072 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437699 5072 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437702 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:40.453398 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437704 5072 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437707 5072 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437709 5072 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437712 5072 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437714 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437716 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437719 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437722 5072 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437724 5072 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437727 5072 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437729 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437733 5072 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437735 5072 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437738 5072 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437741 5072 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437743 5072 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437746 5072 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437748 5072 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437751 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437755 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:40.454276 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437757 5072 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437760 5072 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437764 5072 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437769 5072 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437772 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437774 5072 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437777 5072 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437779 5072 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437782 5072 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437784 5072 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437787 5072 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437789 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437791 5072 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437794 5072 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437797 5072 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437799 5072 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437801 5072 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437804 5072 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437806 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:40.455292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437809 5072 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437813 5072 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437817 5072 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437820 5072 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437824 5072 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437827 5072 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437830 5072 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437833 5072 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437836 5072 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437838 5072 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437841 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437843 5072 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437846 5072 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437848 5072 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437851 5072 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437853 5072 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437856 5072 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437859 5072 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437861 5072 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:40.456345 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437864 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437866 5072 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437868 5072 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437871 5072 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437873 5072 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437876 5072 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437880 5072 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:40.437882 5072 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.437944 5072 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.438914 5072 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.438970 5072 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.439152 5072 server.go:1014] "Starting client certificate rotation" Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.439297 5072 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.439351 5072 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:24:59 +0000 UTC" deadline="2028-07-09 17:15:49.616028502 +0000 UTC" Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.439378 5072 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14843h45m9.176652739s" Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.439641 5072 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:30:40.457181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.440797 5072 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:30:40.457882 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.442068 5072 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:30:40.457882 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.443735 5072 log.go:25] "Validated CRI v1 image API" Oct 30 05:30:40.457882 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.444522 5072 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:30:40.457882 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.446499 5072 fs.go:135] Filesystem UUIDs: map[1b181e71-cc84-4571-a13c-b4605bc09f04:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8f84bfb4-0330-42ed-abbd-b96c7bfae532:/dev/nvme0n1p3] Oct 30 05:30:40.458089 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.446520 5072 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00b270c441897dd0bfe434e21a437881868290a414a56e97e433147e18c2eb5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00b270c441897dd0bfe434e21a437881868290a414a56e97e433147e18c2eb5e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00fddd0a2156689a45838a017d5d90848112f2e7b6ff991cee5c446abeb22b98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00fddd0a2156689a45838a017d5d90848112f2e7b6ff991cee5c446abeb22b98/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02c6460b7041ae4d79c7f5dbf6fbbd60a71d9713ffe867c73db631f1284b457a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02c6460b7041ae4d79c7f5dbf6fbbd60a71d9713ffe867c73db631f1284b457a/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e41d4c791823b5ac233563e3df10bd5abf5b884b016d0c77179feafaf7d1db9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e41d4c791823b5ac233563e3df10bd5abf5b884b016d0c77179feafaf7d1db9/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3363a6b3aad6626ee9f69ab64495d1f09bfb6953a34c5ca365d4132e217f3700/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3363a6b3aad6626ee9f69ab64495d1f09bfb6953a34c5ca365d4132e217f3700/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33f15deada5b62f8a545c185d95583be81f72bf5816b9ac2fc84ce6433ca2153/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33f15deada5b62f8a545c185d95583be81f72bf5816b9ac2fc84ce6433ca2153/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/376c41e33d35a5249bc47fac684bcc0e50fa4b4b7c2fc89cbb1ddfdbd6791608/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/376c41e33d35a5249bc47fac684bcc0e50fa4b4b7c2fc89cbb1ddfdbd6791608/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ef490aca0227af7a5d71c0e102ab90e5c89dc957fe943726430d855908c8893/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ef490aca0227af7a5d71c0e102ab90e5c89dc957fe943726430d855908c8893/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3efc823edb824143ff5be061d45040dbb07f3f226cc5dfb01fc07c77e286852b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3efc823edb824143ff5be061d45040dbb07f3f226cc5dfb01fc07c77e286852b/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/400c440d0d11c896d69540aa9ab0e17271cdf46df7b56b123f20a6e5cda1067f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/400c440d0d11c896d69540aa9ab0e17271cdf46df7b56b123f20a6e5cda1067f/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/407cf142ad3eab13473b854d83ff337fd6ad9d8dfb998eb36353b938b41c87b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/407cf142ad3eab13473b854d83ff337fd6ad9d8dfb998eb36353b938b41c87b6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46d6af6624ec74c3ae1ec215aaaf1c3b9964a093813c1d468a8963329d139b68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46d6af6624ec74c3ae1ec215aaaf1c3b9964a093813c1d468a8963329d139b68/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4844eaf59c98ad95cf2e0b685973a123bd1af956d0d731de12eeeaee3ea1fc11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4844eaf59c98ad95cf2e0b685973a123bd1af956d0d731de12eeeaee3ea1fc11/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4aa810268db76f323f7ed1ed6135c0d43b257fe72cd6acc3d45d57412a328952/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4aa810268db76f323f7ed1ed6135c0d43b257fe72cd6acc3d45d57412a328952/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c76d6cc77a310ed1edf9da56a5c4a3742d5f51753860eabe60dc31345899e48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c76d6cc77a310ed1edf9da56a5c4a3742d5f51753860eabe60dc31345899e48/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c12fb2a72906e947586951e95e8e27f96c1dd35f6c9071342448307434a61e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c12fb2a72906e947586951e95e8e27f96c1dd35f6c9071342448307434a61e1/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/628153e9c7bd40d1735c99a48b322270297393fa436beba06023c088c07c9b4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/628153e9c7bd40d1735c99a48b322270297393fa436beba06023c088c07c9b4c/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e680adc9c3cba9823577b254fd10ef66223c00ece6a78ce082dcb9d97b39d93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e680adc9c3cba9823577b254fd10ef66223c00ece6a78ce082dcb9d97b39d93/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95b5dd0ac769e0a1c9936ffb419279b0ae84622b3c2a5678de83b2d9274d7a2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95b5dd0ac769e0a1c9936ffb419279b0ae84622b3c2a5678de83b2d9274d7a2e/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a83625753bb547479b8cfc9edc087e83052e20c023f8f582568ce55fc0a4bfb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a83625753bb547479b8cfc9edc087e83052e20c023f8f582568ce55fc0a4bfb9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfc163cc70ad703354b5534c2a01765a1b84d0b36b3faa95673a4e97ab17619e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfc163cc70ad703354b5534c2a01765a1b84d0b36b3faa95673a4e97ab17619e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8a892ff18a4642b84b08a0b8919611115d7933c8305a9419ebd8fe783b0b4a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8a892ff18a4642b84b08a0b8919611115d7933c8305a9419ebd8fe783b0b4a1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eaede63d735c2995d5b13d3db51a00c8c303ab805807f5e8d4063c13a6555bd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eaede63d735c2995d5b13d3db51a00c8c303ab805807f5e8d4063c13a6555bd9/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f09805b6662dd398b6fc65a665545903bcf7ff82716dfec8c9e494cf73c83a4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f09805b6662dd398b6fc65a665545903bcf7ff82716dfec8c9e494cf73c83a4a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f34759dd8e5f7cb9a690cf8c0e822e53cf60f041a8291a34b300223e4f21afc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f34759dd8e5f7cb9a690cf8c0e822e53cf60f041a8291a34b300223e4f21afc3/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb7d7b30f6715d9058a8efe8e704cc3bbe1b31e21df705683c085d5eae048859/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb7d7b30f6715d9058a8efe8e704cc3bbe1b31e21df705683c085d5eae048859/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d304ba5-b889-412c-ae7d-b26f04222714/volumes/kubernetes.io~projected/kube-api-access-5c926:{mountpoint:/var/lib/kubelet/pods/0d304ba5-b889-412c-ae7d-b26f04222714/volumes/kubernetes.io~projected/kube-api-access-5c926 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ee3f1bf-665e-4220-ac90-5d0b041d7077/volumes/kubernetes.io~projected/kube-api-access-6ts77:{mountpoint:/var/lib/kubelet/pods/0ee3f1bf-665e-4220-ac90-5d0b041d7077/volumes/kubernetes.io~projected/kube-api-access-6ts77 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12fe3fee-9496-4b7b-8923-d19c585cc4be/volumes/kubernetes.io~projected/kube-api-access-v974p:{mountpoint:/var/lib/kubelet/pods/12fe3fee-9496-4b7b-8923-d19c585cc4be/volumes/kubernetes.io~projected/kube-api-access-v974p major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e52c123-4401-4329-8058-5479f3680281/volumes/kubernetes.io~projected/kube-api-access-kqgpk:{mountpoint:/var/lib/kubelet/pods/1e52c123-4401-4329-8058-5479f3680281/volumes/kubernetes.io~projected/kube-api-access-kqgpk major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26ef5d7f-4915-4cee-a412-c1f1f38410bd/volumes/kubernetes.io~projected/kube-api-access-8wj77:{mountpoint:/var/lib/kubelet/pods/26ef5d7f-4915-4cee-a412-c1f1f38410bd/volumes/kubernetes.io~projected/kube-api-access-8wj77 major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f968039-9ce3-48ea-a1c6-c963206a596c/volumes/kubernetes.io~projected/kube-api-access-bqgr9:{mountpoint:/var/lib/kubelet/pods/2f968039-9ce3-48ea-a1c6-c963206a596c/volumes/kubernetes.io~projected/kube-api-access-bqgr9 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f968039-9ce3-48ea-a1c6-c963206a596c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2f968039-9ce3-48ea-a1c6-c963206a596c/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~projected/kube-api-access-dkzgj:{mountpoint:/var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~projected/kube-api-access-dkzgj major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~projected/kube-api-access-gxd2m:{mountpoint:/var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~projected/kube-api-access-gxd2m major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37/volumes/kubernetes.io~projected/kube-api-access-spgk6:{mountpoint:/var/lib/kubelet/pods/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37/volumes/kubernetes.io~projected/kube-api-access-spgk6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/454eb54a-074f-45e9-94cb-75356bb42e71/volumes/kubernetes.io~projected/kube-api-access-gjbzf:{mountpoint:/var/lib/kubelet/pods/454eb54a-074f-45e9-94cb-75356bb42e71/volumes/kubernetes.io~projected/kube-api-access-gjbzf major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d/volumes/kubernetes.io~projected/kube-api-access-5n589:{mountpoint:/var/lib/kubelet/pods/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d/volumes/kubernetes.io~projected/kube-api-access-5n589 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/510573e7-a380-415f-9609-18470c856703/volumes/kubernetes.io~projected/kube-api-access-48wsv:{mountpoint:/var/lib/kubelet/pods/510573e7-a380-415f-9609-18470c856703/volumes/kubernetes.io~projected/kube-api-access-48wsv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/540c21f9-43da-400c-b0da-e0b01eecedf7/volumes/kubernetes.io~projected/kube-api-access-dgps8:{mountpoint:/var/lib/kubelet/pods/540c21f9-43da-400c-b0da-e0b01eecedf7/volumes/kubernetes.io~projected/kube-api-access-dgps8 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/540c21f9-43da-400c-b0da-e0b01eecedf7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/540c21f9-43da-400c-b0da-e0b01eecedf7/volumes/kubernetes.io~secret/cert major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60d71bc3-a236-4305-baeb-df6403b52f43/volumes/kubernetes.io~projected/kube-api-access-lnkmt:{mountpoint:/var/lib/kubelet/pods/60d71bc3-a236-4305-baeb-df6403b52f43/volumes/kubernetes.io~projected/kube-api-access-lnkmt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60d71bc3-a236-4305-baeb-df6403b52f43/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/60d71bc3-a236-4305-baeb-df6403b52f43/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62348232-2a4d-4ed2-8830-10f547a59fea/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/62348232-2a4d-4ed2-8830-10f547a59fea/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62348232-2a4d-4ed2-8830-10f547a59fea/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/62348232-2a4d-4ed2-8830-10f547a59fea/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62348232-2a4d-4ed2-8830-10f547a59fea/volumes/kubernetes.io~projected/kube-api-access-bjdk9:{mountpoint:/var/lib/kubelet/pods/62348232-2a4d-4ed2-8830-10f547a59fea/volumes/kubernetes.io~projected/kube-api-access-bjdk9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6381b6d8-a45f-43d3-955c-504694db87da/volumes/kubernetes.io~projected/kube-api-access-p4wgj:{mountpoint:/var/lib/kubelet/pods/6381b6d8-a45f-43d3-955c-504694db87da/volumes/kubernetes.io~projected/kube-api-access-p4wgj major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6381b6d8-a45f-43d3-955c-504694db87da/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6381b6d8-a45f-43d3-955c-504694db87da/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6381b6d8-a45f-43d3-955c-504694db87da/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6381b6d8-a45f-43d3-955c-504694db87da/volumes/kubernetes.io~secret/stats-auth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/671efddc-7c80-4be5-827d-0c227985c013/volumes/kubernetes.io~projected/kube-api-access-8hcv9:{mountpoint:/var/lib/kubelet/pods/671efddc-7c80-4be5-827d-0c227985c013/volumes/kubernetes.io~projected/kube-api-access-8hcv9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70464c99-1f1b-49e4-8cc9-335fa1645a98/volumes/kubernetes.io~projected/kube-api-access-rdqgp:{mountpoint:/var/lib/kubelet/pods/70464c99-1f1b-49e4-8cc9-335fa1645a98/volumes/kubernetes.io~projected/kube-api-access-rdqgp major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70464c99-1f1b-49e4-8cc9-335fa1645a98/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/70464c99-1f1b-49e4-8cc9-335fa1645a98/volumes/kubernetes.io~secret/signing-key major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74f707f9-cd75-4d5f-bab3-3ead92d87904/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/74f707f9-cd75-4d5f-bab3-3ead92d87904/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d813b7e-2c08-4c0f-b077-2b529d90cf51/volumes/kubernetes.io~projected/kube-api-access-5kkvb:{mountpoint:/var/lib/kubelet/pods/8d813b7e-2c08-4c0f-b077-2b529d90cf51/volumes/kubernetes.io~projected/kube-api-access-5kkvb major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d813b7e-2c08-4c0f-b077-2b529d90cf51/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8d813b7e-2c08-4c0f-b077-2b529d90cf51/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95b26dcb-4282-498a-b278-4c7bb6ac11a1/volumes/kubernetes.io~projected/kube-api-access-pt7gk:{mountpoint:/var/lib/kubelet/pods/95b26dcb-4282-498a-b278-4c7bb6ac11a1/volumes/kubernetes.io~projected/kube-api-access-pt7gk major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9db8ada4-0aec-4b8d-844b-e7d9abbe0850/volumes/kubernetes.io~projected/kube-api-access-n78ks:{mountpoint:/var/lib/kubelet/pods/9db8ada4-0aec-4b8d-844b-e7d9abbe0850/volumes/kubernetes.io~projected/kube-api-access-n78ks major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e2abeba-b181-40af-89c2-c3782ad160c5/volumes/kubernetes.io~projected/kube-api-access-vjx5n:{mountpoint:/var/lib/kubelet/pods/9e2abeba-b181-40af-89c2-c3782ad160c5/volumes/kubernetes.io~projected/kube-api-access-vjx5n major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae0a00ed-126f-4c35-a159-aa6ed7e2c72f/volumes/kubernetes.io~projected/kube-api-access-h5zrt:{mountpoint:/var/lib/kubelet/pods/ae0a00ed-126f-4c35-a159-aa6ed7e2c72f/volumes/kubernetes.io~projected/kube-api-access-h5zrt major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af48072f-2e02-46db-9b2d-7eb2da4cd2e4/volumes/kubernetes.io~projected/kube-api-access-nhq8c:{mountpoint:/var/lib/kubelet/pods/af48072f-2e02-46db-9b2d-7eb2da4cd2e4/volumes/kubernetes.io~projected/kube-api-access-nhq8c major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afe5578f-4885-4a55-a947-4c9bfe331614/volumes/kubernetes.io~projected/kube-api-access-wmfrm:{mountpoint:/var/lib/kubelet/pods/afe5578f-4885-4a55-a947-4c9bfe331614/volumes/kubernetes.io~projected/kube-api-access-wmfrm major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9d914f2-a350-49db-b937-c4774be4415e/volumes/kubernetes.io~projected/kube-api-access-5kstz:{mountpoint:/var/lib/kubelet/pods/b9d914f2-a350-49db-b937-c4774be4415e/volumes/kubernetes.io~projected/kube-api-access-5kstz major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3/volumes/kubernetes.io~projected/kube-api-access-ttx6t:{mountpoint:/var/lib/kubelet/pods/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3/volumes/kubernetes.io~projected/kube-api-access-ttx6t major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0f89396-3754-4034-af10-b9a951d3bbb1/volumes/kubernetes.io~projected/kube-api-access-xddvn:{mountpoint:/var/lib/kubelet/pods/d0f89396-3754-4034-af10-b9a951d3bbb1/volumes/kubernetes.io~projected/kube-api-access-xddvn major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0a3b194-c453-404f-92a4-0b1ab87f7699/volumes/kubernetes.io~projected/kube-api-access-v6f78:{mountpoint:/var/lib/kubelet/pods/e0a3b194-c453-404f-92a4-0b1ab87f7699/volumes/kubernetes.io~projected/kube-api-access-v6f78 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f25962bc-3b24-4f15-842f-e3d9771a1809/volumes/kubernetes.io~projected/kube-api-access-g7khr:{mountpoint:/var/lib/kubelet/pods/f25962bc-3b24-4f15-842f-e3d9771a1809/volumes/kubernetes.io~projected/kube-api-access-g7khr major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3d3980e-8842-4c87-a275-8bf0da706fba/volumes/kubernetes.io~projected/kube-api-access-l7q66:{mountpoint:/var/lib/kubelet/pods/f3d3980e-8842-4c87-a275-8bf0da706fba/volumes/kubernetes.io~projected/kube-api-access-l7q66 major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4f50efbfc879ed0b5eed21acd3f6365a534ea8b3fd812508cd65549bc27a6c64/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b154646cfaacef412e4ae1938faa8574df47401c04397cdef5170320da29eafb/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4216365859eca43ed0551d1817f1f913bedaa111e4d1f9f7b54d34c85cba6b7a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0ff783e6bcbc5e93f585a7f13ef0048262b181ffdb42befaa9185afb192ecb13/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fc8590f5a5ae2bf265a8d7ee3410af0339d0beb17fb3e453a75e4ff7e0321301/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e80ac47924be7d53beba161323193792e74957da3016214211819a83aba50d11/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/53b92d6bc6b3dc99dcc1582249728857bf03f80431d3d033cd1a5784ec698d59/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/61ed4d44f83ce7aeb50b1f7e96309f8aad903b7edf49c83ab339c00ef9e4573d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/4dfc77cf9c7719232de49bc5f1aa518b5af05c8b8dcb6199cf5997bd1e70ad4f/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/91c51511d9738019046587e2f5d57cd55401e61d489b50e71dba21efa97af871/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e4c5d6a17ab36ae67d3848e31799ff01e31363d756f9329c2950c8b9bbbb34fd/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b790049ea79975886d05ceda22a8cf272505d5f76574d257df3f55c000314c69/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/db385cdc9b57497d5c7a7e1e8e750ef96eed86ec695e60f848e0e8651f566a5b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/545bc1c0011ecb19f44e0f98de881024c43d0f0de11af5db7ad1b048a4ffb8eb/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8654f3e712e4785755d863b37559acd1ce9072fbe998c7ba0e46404c294292e0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6dc8f0fdfa149b4ca8de265419332c5741a596978a668e0631090370e0a6f094/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0ef5722105ae6195c58d3885bc7d20d14bc4e0c163ecf62a20e2d542a849c964/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/36beb4ddf974947d4480760812be5e4c24f43ff4d76d9b999074eac05cc5b00f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8a7dc1e8fa37bc60d1bbf50b118b50f5377dbcc6253ae4eb56e29b1bed0411bc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/055cf5856e39b00fd1271686ee6ff9ff7670cf2498222abe82171b9e38f26b99/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d249568a76be2db7395b641ab077e7c5fd7031bd8b3eb50559e822f410c436a5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/223545e8eeeb4e2955d7667baf204ae5e282756f0852e3143d88074cd1b5f390/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/f49cafd0671ed5304d1a6393449e1a1ae999301a4f6905f24ba64054f7619865/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/c7b9882ef9678b2b55046d95c6cc623e73c9e3c01a16c2777243b757ec8e1434/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/fd4d4a6a66b37d3977db81528b3a28cfbb97b0ef5c20d35e114ba4980af9663a/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/6c3fda4947be8e3889182ca81a11ad7d9b5bc7e56a1c728ea69ca2e8faea4eae/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/487c7f721e283242bb675613328140b37f1b262ef73cb33fdd2652d5045fc076/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/9c24a8fb19869ae2413055ba7ea33e451b805e93d958597fb25dd02e00c529fe/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/9ea487895e48deed5404d30f6ef104b059815e4ec48d7957f7fa9e9966b7685f/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ff7c07b51e874ce11ad42281c9814cabc0dc52eb494719e71b733568899462ac/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/acaac38e54b74d864f1a38037bde23c4ef9f061451368effce6030f45efb97f3/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/932bac5bca7a360161ad71d8af31f1c5b0a10423ccc25dacf0157cb2e75f0b00/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/6ce2d992356f20396689d869dee336e09a09e3f9796efc1578d259f98aba6e24/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/dfecfd0fad38db340c9a7adbdcee00f74d96e9f76c02f62b5562891f0066a9bb/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/a861aeef134d4e023db1d876bf100734326f73b743cb953a4e033fd3dd3dfe15/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/a7a880931c422c5665c61df80bea6656a56c8c871c5537f026c350e3d1bda48d/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/57513e59ae393ff43595b1fc640b7cd2064a95e0e21cba8f71f300c27fe9d8d2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/c6b28da5f7c8e6064d7f84099b85150a845012d899b5ddfce8c8619beca466cb/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/0fa9101980e76641ab2ff0950f0bb4c968df80a19fa9c071c68ef5e86999378d/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/f1cafc788c297013cd8c8eb46315bbfa5689a4060bfc13b7fb3e404768e78419/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/446e65c9868a49ae006df32f82feca6b218ed9da711e0160d26d956887dec85c/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/48e344a0fca3f44e330c7bd2e7a51e4866da86df0db283a4eb696868f46d8afa/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/99342a5db514830eabde9e15b2a83d1566ffb5f3a64ffd471e19194922115e97/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/011004988f76e2b04dad067246052d4fcba1a702b84539b940eb806ed15c32e6/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/777540d8181d31f0c76d8d6365c19420eccb28b5da5a121987efe9825e257659/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/22acf4990ca1e9aa3eab6cae2c45d244b9c5b7e75b14ccdccf0ffb8a77f8af88/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/801ab7dff62a7e25f6de004b4119bbebb204c075e4de30f4eb510e42a4bf835b/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/c80db77d11b721f08cda0e722c681d402f7ba8f0e39c78f3afa4b42845cc3cb2/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/9e9ec180e17009cd868d3e9709d1adaa3494378b168fb45c9f345813a48feae5/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/040c3187b7eeb1a5523117a4683522c05ba3bb095593d285265ec5aa09be6d81/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/5b914f84b3b518415c778320cbfdb7eb37f8b945b22ed5c51a1593dde5274d2d/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/a8469b7b83403e726ac4ff78a969a9bff13be9abe9a0f2b61915fa92c6bc049a/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/6b7abbf93611d02eb55459d699d6c62585f0eb6d93b8f826a2e2d5c1f982dee2/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/bdf193cafa8001a5d2fe20e8b1567fefbca989b151597d31e69f90aa1a74d1dd/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/1f5e3a8bf186cabfe267ea0dafb4bc789968d3c15205f1c6d33954dedce14aed/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/aa5370724e07cace178ac6efb943442ecc84c7d1be4013b2a192d5bd729535d0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/34fa874f813f2d14ca7abe568ae1cf10cbb8174da9cd3ae30aa9611fc8f01680/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/d678e9543e5da49eba45d58781c16c29752d420016961c040fb867bc7f61e781/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/a75fa389eea6aa80bf4db91a2629963bce7dbbd01b5c6bb0e708ed6ff11425d2/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/82e0ac19460548d8eef2337e581915e47afbb8ad9c58be72b1d0fe145a878698/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f36edb609ea8c398a95c19cb00b62cba7cf19eaa5915ec7d276062fa6d5087c7/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/432f0941161a0939849aa340a71d3061cb15d5e77151767bbcf9c8f9b5ce5d4e/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 05:30:40.458316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.457207 5072 manager.go:217] Machine: {Timestamp:2026-10-30 05:30:40.456386259 +0000 UTC m=+0.060075511 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099750 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a0b59bdbaf143c2eeda045376a42 SystemUUID:ec28a0b5-9bdb-af14-3c2e-eda045376a42 BootID:dcc054e7-04fe-4fc1-829b-13d4bd128279 Filesystems:[{Device:/run/containers/storage/overlay-containers/407cf142ad3eab13473b854d83ff337fd6ad9d8dfb998eb36353b938b41c87b6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/628153e9c7bd40d1735c99a48b322270297393fa436beba06023c088c07c9b4c/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f25962bc-3b24-4f15-842f-e3d9771a1809/volumes/kubernetes.io~projected/kube-api-access-g7khr DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1e52c123-4401-4329-8058-5479f3680281/volumes/kubernetes.io~projected/kube-api-access-kqgpk DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/60d71bc3-a236-4305-baeb-df6403b52f43/volumes/kubernetes.io~projected/kube-api-access-lnkmt DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eaede63d735c2995d5b13d3db51a00c8c303ab805807f5e8d4063c13a6555bd9/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00fddd0a2156689a45838a017d5d90848112f2e7b6ff991cee5c446abeb22b98/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/540c21f9-43da-400c-b0da-e0b01eecedf7/volumes/kubernetes.io~projected/kube-api-access-dgps8 DeviceMajor:0 DeviceMinor:237 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ef490aca0227af7a5d71c0e102ab90e5c89dc957fe943726430d855908c8893/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/62348232-2a4d-4ed2-8830-10f547a59fea/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/12fe3fee-9496-4b7b-8923-d19c585cc4be/volumes/kubernetes.io~projected/kube-api-access-v974p DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f09805b6662dd398b6fc65a665545903bcf7ff82716dfec8c9e494cf73c83a4a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33f15deada5b62f8a545c185d95583be81f72bf5816b9ac2fc84ce6433ca2153/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70464c99-1f1b-49e4-8cc9-335fa1645a98/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4844eaf59c98ad95cf2e0b685973a123bd1af956d0d731de12eeeaee3ea1fc11/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e680adc9c3cba9823577b254fd10ef66223c00ece6a78ce082dcb9d97b39d93/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d813b7e-2c08-4c0f-b077-2b529d90cf51/volumes/kubernetes.io~projected/kube-api-access-5kkvb DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4aa810268db76f323f7ed1ed6135c0d43b257fe72cd6acc3d45d57412a328952/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8d813b7e-2c08-4c0f-b077-2b529d90cf51/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95b26dcb-4282-498a-b278-4c7bb6ac11a1/volumes/kubernetes.io~projected/kube-api-access-pt7gk DeviceMajor:0 DeviceMinor:334 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46d6af6624ec74c3ae1ec215aaaf1c3b9964a093813c1d468a8963329d139b68/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/400c440d0d11c896d69540aa9ab0e17271cdf46df7b56b123f20a6e5cda1067f/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/376c41e33d35a5249bc47fac684bcc0e50fa4b4b7c2fc89cbb1ddfdbd6791608/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/74f707f9-cd75-4d5f-bab3-3ead92d87904/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfc163cc70ad703354b5534c2a01765a1b84d0b36b3faa95673a4e97ab17619e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/62348232-2a4d-4ed2-8830-10f547a59fea/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f34759dd8e5f7cb9a690cf8c0e822e53cf60f041a8291a34b300223e4f21afc3/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~projected/kube-api-access-gxd2m DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b9d914f2-a350-49db-b937-c4774be4415e/volumes/kubernetes.io~projected/kube-api-access-5kstz DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb7d7b30f6715d9058a8efe8e704cc3bbe1b31e21df705683c085d5eae048859/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00b270c441897dd0bfe434e21a437881868290a414a56e97e433147e18c2eb5e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/62348232-2a4d-4ed2-8830-10f547a59fea/volumes/kubernetes.io~projected/kube-api-access-bjdk9 DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e41d4c791823b5ac233563e3df10bd5abf5b884b016d0c77179feafaf7d1db9/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/671efddc-7c80-4be5-827d-0c227985c013/volumes/kubernetes.io~projected/kube-api-access-8hcv9 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d/volumes/kubernetes.io~projected/kube-api-access-5n589 DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02c6460b7041ae4d79c7f5dbf6fbbd60a71d9713ffe867c73db631f1284b457a/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6381b6d8-a45f-43d3-955c-504694db87da/volumes/kubernetes.io~projected/kube-api-access-p4wgj DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e2abeba-b181-40af-89c2-c3782ad160c5/volumes/kubernetes.io~projected/kube-api-access-vjx5n DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0ee3f1bf-665e-4220-ac90-5d0b041d7077/volumes/kubernetes.io~projected/kube-api-access-6ts77 DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70464c99-1f1b-49e4-8cc9-335fa1645a98/volumes/kubernetes.io~projected/kube-api-access-rdqgp DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3efc823edb824143ff5be061d45040dbb07f3f226cc5dfb01fc07c77e286852b/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3363a6b3aad6626ee9f69ab64495d1f09bfb6953a34c5ca365d4132e217f3700/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3/volumes/kubernetes.io~projected/kube-api-access-ttx6t DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2f968039-9ce3-48ea-a1c6-c963206a596c/volumes/kubernetes.io~projected/kube-api-access-bqgr9 DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/510573e7-a380-415f-9609-18470c856703/volumes/kubernetes.io~projected/kube-api-access-48wsv DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/454eb54a-074f-45e9-94cb-75356bb42e71/volumes/kubernetes.io~projected/kube-api-access-gjbzf DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c76d6cc77a310ed1edf9da56a5c4a3742d5f51753860eabe60dc31345899e48/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~projected/kube-api-access-dkzgj DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a83625753bb547479b8cfc9edc087e83052e20c023f8f582568ce55fc0a4bfb9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6381b6d8-a45f-43d3-955c-504694db87da/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95b5dd0ac769e0a1c9936ffb419279b0ae84622b3c2a5678de83b2d9274d7a2e/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37/volumes/kubernetes.io~projected/kube-api-access-spgk6 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af48072f-2e02-46db-9b2d-7eb2da4cd2e4/volumes/kubernetes.io~projected/kube-api-access-nhq8c DeviceMajor:0 DeviceMinor:333 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9db8ada4-0aec-4b8d-844b-e7d9abbe0850/volumes/kubernetes.io~projected/kube-api-access-n78ks DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26ef5d7f-4915-4cee-a412-c1f1f38410bd/volumes/kubernetes.io~projected/kube-api-access-8wj77 DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0a3b194-c453-404f-92a4-0b1ab87f7699/volumes/kubernetes.io~projected/kube-api-access-v6f78 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3d3980e-8842-4c87-a275-8bf0da706fba/volumes/kubernetes.io~projected/kube-api-access-l7q66 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0d304ba5-b889-412c-ae7d-b26f04222714/volumes/kubernetes.io~projected/kube-api-access-5c926 DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60d71bc3-a236-4305-baeb-df6403b52f43/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/540c21f9-43da-400c-b0da-e0b01eecedf7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c12fb2a72906e947586951e95e8e27f96c1dd35f6c9071342448307434a61e1/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:268 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6381b6d8-a45f-43d3-955c-504694db87da/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/afe5578f-4885-4a55-a947-4c9bfe331614/volumes/kubernetes.io~projected/kube-api-access-wmfrm DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0f89396-3754-4034-af10-b9a951d3bbb1/volumes/kubernetes.io~projected/kube-api-access-xddvn DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f968039-9ce3-48ea-a1c6-c963206a596c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae0a00ed-126f-4c35-a159-aa6ed7e2c72f/volumes/kubernetes.io~projected/kube-api-access-h5zrt DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8a892ff18a4642b84b08a0b8919611115d7933c8305a9419ebd8fe783b0b4a1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00fddd0a2156689 MacAddress:5a:c9:83:cc:65:96 Speed:10000 Mtu:8901} {Name:02c6460b7041ae4 MacAddress:4a:bf:0f:10:9c:57 Speed:10000 Mtu:8901} {Name:0e41d4c791823b5 MacAddress:3e:2b:b4:a3:1a:62 Speed:10000 Mtu:8901} {Name:3363a6b3aad6626 MacAddress:e2:31:82:a6:20:25 Speed:10000 Mtu:8901} {Name:33f15deada5b62f MacAddress:02:93:12:d0:1a:3d Speed:10000 Mtu:8901} {Name:376c41e33d35a52 MacAddress:6a:59:9c:34:0b:43 Speed:10000 Mtu:8901} {Name:3ef490aca0227af MacAddress:7a:64:cd:49:e2:1f Speed:10000 Mtu:8901} {Name:3efc823edb82414 MacAddress:12:a3:5a:8c:cf:62 Speed:10000 Mtu:8901} {Name:400c440d0d11c89 MacAddress:62:8f:ab:d0:3b:b8 Speed:10000 Mtu:8901} {Name:46d6af6624ec74c MacAddress:d6:6b:1c:fb:53:41 Speed:10000 Mtu:8901} {Name:4aa810268db76f3 MacAddress:fa:d0:ca:e9:b0:ec Speed:10000 Mtu:8901} {Name:4c76d6cc77a310e MacAddress:66:92:80:d8:38:c0 Speed:10000 Mtu:8901} {Name:5c12fb2a72906e9 MacAddress:b2:bc:d2:88:26:ad Speed:10000 Mtu:8901} {Name:628153e9c7bd40d MacAddress:c6:fb:4f:f6:64:e9 Speed:10000 Mtu:8901} {Name:95b5dd0ac769e0a MacAddress:5a:be:3a:bf:94:2b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e3:97:4d:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:de:2f:47:29:e3:d9 Speed:0 Mtu:8901} {Name:eaede63d735c299 MacAddress:92:7f:6e:30:5f:ee Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e3:97:4d:7d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7a:ea:22:c6:02:3a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:2c:02:3f:ef:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:30:40.458316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.457947 5072 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:30:40.458316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458014 5072 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:30:40.458577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458318 5072 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:30:40.458577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458342 5072 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:30:40.458577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458496 5072 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:30:40.458577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458504 5072 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:30:40.458577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458515 5072 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:30:40.458577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458530 5072 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:30:40.458577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458549 5072 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:30:40.458792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458703 5072 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:30:40.458823 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458799 5072 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:30:40.458871 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458862 5072 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:30:40.458916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458876 5072 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:30:40.458916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458889 5072 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:30:40.458916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458902 5072 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:30:40.458916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.458913 5072 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:30:40.459740 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.459724 5072 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:30:40.459740 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.459738 5072 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:30:40.460313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.460292 5072 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:30:40.462471 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462445 5072 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:30:40.462526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462475 5072 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:30:40.462594 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462584 5072 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 05:30:40.462779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462768 5072 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:30:40.462808 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462798 5072 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:30:40.462894 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462885 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:30:40.462923 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462905 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:30:40.462923 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462914 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:30:40.462976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462924 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:30:40.462976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462932 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:30:40.462976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462938 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:30:40.462976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462949 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:30:40.462976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462955 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:30:40.462976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462961 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:30:40.462976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462967 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:30:40.463191 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462980 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:30:40.463191 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.462990 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:30:40.463191 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.463018 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:30:40.463191 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.463027 5072 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:30:40.466647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.466632 5072 server.go:1263] "Started kubelet" Oct 30 05:30:40.467741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.466948 5072 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:30:40.467741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.466943 5072 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:30:40.467741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.467031 5072 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:30:40.467741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.467330 5072 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:30:40.467147 ip-10-0-138-200 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:30:40.469977 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.469743 5072 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:30:40.472327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.472300 5072 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:30:40.472418 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.472332 5072 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:30:40.472418 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.472380 5072 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:25:01 +0000 UTC" deadline="2028-04-07 05:17:01.113559844 +0000 UTC" Oct 30 05:30:40.472418 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.472398 5072 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12599h46m20.641164825s" Oct 30 05:30:40.474331 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:40.474305 5072 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found" Oct 30 05:30:40.474500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.474484 5072 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:30:40.474587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.474502 5072 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:30:40.474587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.474485 5072 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:30:40.475805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.475785 5072 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:30:40.475805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.475806 5072 factory.go:55] Registering systemd factory Oct 30 05:30:40.475988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.475816 5072 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:30:40.476060 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.476032 5072 factory.go:153] Registering CRI-O factory Oct 30 05:30:40.476060 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.476045 5072 factory.go:223] Registration of the crio container factory successfully Oct 30 05:30:40.476060 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.476062 5072 factory.go:103] Registering Raw factory Oct 30 05:30:40.476215 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.476078 5072 manager.go:1199] Started watching for new ooms in manager Oct 30 05:30:40.476509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.476495 5072 manager.go:319] Starting recovery of all containers Oct 30 05:30:40.477347 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477310 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26ef5d7f-4915-4cee-a412-c1f1f38410bd" volumeName="kubernetes.io/projected/26ef5d7f-4915-4cee-a412-c1f1f38410bd-kube-api-access-8wj77" seLinuxMountContext="" Oct 30 05:30:40.477429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477352 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd827f0-3153-4b94-8326-8c6fd0744f6b" volumeName="kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj" seLinuxMountContext="" Oct 30 05:30:40.477429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477367 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd827f0-3153-4b94-8326-8c6fd0744f6b" volumeName="kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration" seLinuxMountContext="" Oct 30 05:30:40.477429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477379 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d" volumeName="kubernetes.io/secret/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-serving-cert" seLinuxMountContext="" Oct 30 05:30:40.477429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477393 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="540c21f9-43da-400c-b0da-e0b01eecedf7" volumeName="kubernetes.io/projected/540c21f9-43da-400c-b0da-e0b01eecedf7-kube-api-access-dgps8" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477439 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="540c21f9-43da-400c-b0da-e0b01eecedf7" volumeName="kubernetes.io/secret/540c21f9-43da-400c-b0da-e0b01eecedf7-cert" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477452 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a3b194-c453-404f-92a4-0b1ab87f7699" volumeName="kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-cni-binary-copy" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477481 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d3980e-8842-4c87-a275-8bf0da706fba" volumeName="kubernetes.io/projected/f3d3980e-8842-4c87-a275-8bf0da706fba-kube-api-access-l7q66" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477494 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f968039-9ce3-48ea-a1c6-c963206a596c" volumeName="kubernetes.io/projected/2f968039-9ce3-48ea-a1c6-c963206a596c-kube-api-access-bqgr9" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477505 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35863a55-5422-4304-ad71-afddab4d4807" volumeName="kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477516 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37" volumeName="kubernetes.io/configmap/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-serviceca" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477526 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="429c555a-f735-446f-8577-4f7670a5d094" volumeName="kubernetes.io/configmap/429c555a-f735-446f-8577-4f7670a5d094-nginx-conf" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477538 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d" volumeName="kubernetes.io/configmap/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-config" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477551 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62348232-2a4d-4ed2-8830-10f547a59fea" volumeName="kubernetes.io/projected/62348232-2a4d-4ed2-8830-10f547a59fea-kube-api-access-bjdk9" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477566 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f25962bc-3b24-4f15-842f-e3d9771a1809" volumeName="kubernetes.io/projected/f25962bc-3b24-4f15-842f-e3d9771a1809-kube-api-access-g7khr" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477577 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd827f0-3153-4b94-8326-8c6fd0744f6b" volumeName="kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477590 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f968039-9ce3-48ea-a1c6-c963206a596c" volumeName="kubernetes.io/secret/2f968039-9ce3-48ea-a1c6-c963206a596c-serving-cert" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477601 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35863a55-5422-4304-ad71-afddab4d4807" volumeName="kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477617 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="454eb54a-074f-45e9-94cb-75356bb42e71" volumeName="kubernetes.io/projected/454eb54a-074f-45e9-94cb-75356bb42e71-kube-api-access-gjbzf" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477627 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d" volumeName="kubernetes.io/projected/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-kube-api-access-5n589" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477638 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d71bc3-a236-4305-baeb-df6403b52f43" volumeName="kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-service-ca-bundle" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477650 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db8ada4-0aec-4b8d-844b-e7d9abbe0850" volumeName="kubernetes.io/projected/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-kube-api-access-n78ks" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477661 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35863a55-5422-4304-ad71-afddab4d4807" volumeName="kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477672 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510573e7-a380-415f-9609-18470c856703" volumeName="kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477685 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d71bc3-a236-4305-baeb-df6403b52f43" volumeName="kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-snapshots" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477697 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6381b6d8-a45f-43d3-955c-504694db87da" volumeName="kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-stats-auth" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477708 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f707f9-cd75-4d5f-bab3-3ead92d87904" volumeName="kubernetes.io/secret/74f707f9-cd75-4d5f-bab3-3ead92d87904-agent-certs" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477734 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a3b194-c453-404f-92a4-0b1ab87f7699" volumeName="kubernetes.io/projected/e0a3b194-c453-404f-92a4-0b1ab87f7699-kube-api-access-v6f78" seLinuxMountContext="" Oct 30 05:30:40.477739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477754 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd827f0-3153-4b94-8326-8c6fd0744f6b" volumeName="kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477767 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ee3f1bf-665e-4220-ac90-5d0b041d7077" volumeName="kubernetes.io/projected/0ee3f1bf-665e-4220-ac90-5d0b041d7077-kube-api-access-6ts77" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477778 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12fe3fee-9496-4b7b-8923-d19c585cc4be" volumeName="kubernetes.io/projected/12fe3fee-9496-4b7b-8923-d19c585cc4be-kube-api-access-v974p" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477792 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd827f0-3153-4b94-8326-8c6fd0744f6b" volumeName="kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477806 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d71bc3-a236-4305-baeb-df6403b52f43" volumeName="kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-tmp" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477845 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6381b6d8-a45f-43d3-955c-504694db87da" volumeName="kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-default-certificate" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477860 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95b26dcb-4282-498a-b278-4c7bb6ac11a1" volumeName="kubernetes.io/projected/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-api-access-pt7gk" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477874 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3" volumeName="kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-script-lib" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477887 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d71bc3-a236-4305-baeb-df6403b52f43" volumeName="kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477899 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d813b7e-2c08-4c0f-b077-2b529d90cf51" volumeName="kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-config" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477912 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d813b7e-2c08-4c0f-b077-2b529d90cf51" volumeName="kubernetes.io/projected/8d813b7e-2c08-4c0f-b077-2b529d90cf51-kube-api-access-5kkvb" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477924 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d914f2-a350-49db-b937-c4774be4415e" volumeName="kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477936 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f89396-3754-4034-af10-b9a951d3bbb1" volumeName="kubernetes.io/projected/d0f89396-3754-4034-af10-b9a951d3bbb1-kube-api-access-xddvn" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477949 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e52c123-4401-4329-8058-5479f3680281" volumeName="kubernetes.io/configmap/1e52c123-4401-4329-8058-5479f3680281-samples-operator-config" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477961 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35863a55-5422-4304-ad71-afddab4d4807" volumeName="kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477974 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f707f9-cd75-4d5f-bab3-3ead92d87904" volumeName="kubernetes.io/configmap/74f707f9-cd75-4d5f-bab3-3ead92d87904-konnectivity-ca" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477986 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d914f2-a350-49db-b937-c4774be4415e" volumeName="kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.477997 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f968039-9ce3-48ea-a1c6-c963206a596c" volumeName="kubernetes.io/configmap/2f968039-9ce3-48ea-a1c6-c963206a596c-config" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478011 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510573e7-a380-415f-9609-18470c856703" volumeName="kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-binary-copy" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478022 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d813b7e-2c08-4c0f-b077-2b529d90cf51" volumeName="kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-trusted-ca" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478033 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db8ada4-0aec-4b8d-844b-e7d9abbe0850" volumeName="kubernetes.io/configmap/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-iptables-alerter-script" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478045 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afe5578f-4885-4a55-a947-4c9bfe331614" volumeName="kubernetes.io/projected/afe5578f-4885-4a55-a947-4c9bfe331614-kube-api-access-wmfrm" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478056 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a3b194-c453-404f-92a4-0b1ab87f7699" volumeName="kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-daemon-config" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478068 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e52c123-4401-4329-8058-5479f3680281" volumeName="kubernetes.io/projected/1e52c123-4401-4329-8058-5479f3680281-kube-api-access-kqgpk" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478081 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d914f2-a350-49db-b937-c4774be4415e" volumeName="kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478093 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3" volumeName="kubernetes.io/secret/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478104 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d3980e-8842-4c87-a275-8bf0da706fba" volumeName="kubernetes.io/empty-dir/f3d3980e-8842-4c87-a275-8bf0da706fba-tmp-dir" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478116 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70464c99-1f1b-49e4-8cc9-335fa1645a98" volumeName="kubernetes.io/secret/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-key" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478137 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d914f2-a350-49db-b937-c4774be4415e" volumeName="kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478149 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3" volumeName="kubernetes.io/projected/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-kube-api-access-ttx6t" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478164 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35863a55-5422-4304-ad71-afddab4d4807" volumeName="kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478174 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62348232-2a4d-4ed2-8830-10f547a59fea" volumeName="kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-tmp" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478186 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95b26dcb-4282-498a-b278-4c7bb6ac11a1" volumeName="kubernetes.io/empty-dir/95b26dcb-4282-498a-b278-4c7bb6ac11a1-data-volume" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478198 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afe5578f-4885-4a55-a947-4c9bfe331614" volumeName="kubernetes.io/configmap/afe5578f-4885-4a55-a947-4c9bfe331614-telemetry-config" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478210 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3" volumeName="kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-env-overrides" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478225 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ee3f1bf-665e-4220-ac90-5d0b041d7077" volumeName="kubernetes.io/empty-dir/0ee3f1bf-665e-4220-ac90-5d0b041d7077-tmp-dir" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478236 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35863a55-5422-4304-ad71-afddab4d4807" volumeName="kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478247 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510573e7-a380-415f-9609-18470c856703" volumeName="kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478259 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d71bc3-a236-4305-baeb-df6403b52f43" volumeName="kubernetes.io/projected/60d71bc3-a236-4305-baeb-df6403b52f43-kube-api-access-lnkmt" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478270 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62348232-2a4d-4ed2-8830-10f547a59fea" volumeName="kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-etc-tuned" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478280 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6381b6d8-a45f-43d3-955c-504694db87da" volumeName="kubernetes.io/projected/6381b6d8-a45f-43d3-955c-504694db87da-kube-api-access-p4wgj" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478293 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70464c99-1f1b-49e4-8cc9-335fa1645a98" volumeName="kubernetes.io/configmap/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-cabundle" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478306 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70464c99-1f1b-49e4-8cc9-335fa1645a98" volumeName="kubernetes.io/projected/70464c99-1f1b-49e4-8cc9-335fa1645a98-kube-api-access-rdqgp" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478317 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35863a55-5422-4304-ad71-afddab4d4807" volumeName="kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478328 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37" volumeName="kubernetes.io/projected/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-kube-api-access-spgk6" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478340 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ee3f1bf-665e-4220-ac90-5d0b041d7077" volumeName="kubernetes.io/configmap/0ee3f1bf-665e-4220-ac90-5d0b041d7077-config-volume" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478352 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde" volumeName="kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478365 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae0a00ed-126f-4c35-a159-aa6ed7e2c72f" volumeName="kubernetes.io/projected/ae0a00ed-126f-4c35-a159-aa6ed7e2c72f-kube-api-access-h5zrt" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478376 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3" volumeName="kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-config" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478388 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd827f0-3153-4b94-8326-8c6fd0744f6b" volumeName="kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478400 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd827f0-3153-4b94-8326-8c6fd0744f6b" volumeName="kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478412 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d71bc3-a236-4305-baeb-df6403b52f43" volumeName="kubernetes.io/secret/60d71bc3-a236-4305-baeb-df6403b52f43-serving-cert" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478425 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="671efddc-7c80-4be5-827d-0c227985c013" volumeName="kubernetes.io/projected/671efddc-7c80-4be5-827d-0c227985c013-kube-api-access-8hcv9" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478436 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d813b7e-2c08-4c0f-b077-2b529d90cf51" volumeName="kubernetes.io/secret/8d813b7e-2c08-4c0f-b077-2b529d90cf51-serving-cert" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478447 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d304ba5-b889-412c-ae7d-b26f04222714" volumeName="kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478476 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="510573e7-a380-415f-9609-18470c856703" volumeName="kubernetes.io/projected/510573e7-a380-415f-9609-18470c856703-kube-api-access-48wsv" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478490 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e2abeba-b181-40af-89c2-c3782ad160c5" volumeName="kubernetes.io/projected/9e2abeba-b181-40af-89c2-c3782ad160c5-kube-api-access-vjx5n" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478501 5072 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af48072f-2e02-46db-9b2d-7eb2da4cd2e4" volumeName="kubernetes.io/projected/af48072f-2e02-46db-9b2d-7eb2da4cd2e4-kube-api-access-nhq8c" seLinuxMountContext="" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478512 5072 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:30:40.478856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.478520 5072 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:30:40.498266 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.498231 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:30:40.499111 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.499090 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:30:40.508210 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.508168 5072 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:30:40.509979 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.509694 5072 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:30:40.509979 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.509719 5072 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:30:40.509979 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.509741 5072 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:30:40.509979 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:40.509780 5072 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 05:30:40.524423 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.524396 5072 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 05:30:40.525599 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.525575 5072 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:40.528607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.528582 5072 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="012aaef3e7f70d4648a128d38fa0bb440b1b5ad129eee161ec71ae6fcc51b834" exitCode=0 Oct 30 05:30:40.528607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.528603 5072 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="a2b7926adcb43a8ab8c2622dc7bce4755fe7b79c6842fc93e0a3ad79f820b2ab" exitCode=0 Oct 30 05:30:40.528607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.528609 5072 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="1f7ff4920c9e8eaccf8f23ebcd9f0756b838d410ab6dbb707e1d36f031f967ef" exitCode=0 Oct 30 05:30:40.528815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.528614 5072 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="4cb1284e6667520b5ee716cfd51ec96ba154f7c6544f55171b8a70b5a881d86a" exitCode=0 Oct 30 05:30:40.528815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.528620 5072 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="1511facb36682665b29767d9c47a66bc4b5b4c09373437fc90bc34a2f382cd13" exitCode=0 Oct 30 05:30:40.528815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.528625 5072 generic.go:356] "Generic (PLEG): container finished" podID="510573e7-a380-415f-9609-18470c856703" containerID="ac58814d9f28f586886d10d2e6f9c2d4d06836dcd7c4b55ea499f1256bf6d8f9" exitCode=0 Oct 30 05:30:40.536906 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.536882 5072 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:40.538435 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.538405 5072 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:40.538802 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.538780 5072 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:40.542010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.541992 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:30:40.542095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.542025 5072 generic.go:356] "Generic (PLEG): container finished" podID="8d813b7e-2c08-4c0f-b077-2b529d90cf51" containerID="76737ca3daa3bc1b7826bbb28c96768c103e4eb44fcfedc551d1ee20fc521585" exitCode=255 Oct 30 05:30:40.551342 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.551282 5072 generic.go:356] "Generic (PLEG): container finished" podID="77d685204da0cc5fa69ed201b4cfe0a4" containerID="d24f14bcbd1e4d496f361468e715510fe2777a9870a4cb50df5103ac557bf6ec" exitCode=0 Oct 30 05:30:40.581011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.580987 5072 manager.go:324] Recovery completed Oct 30 05:30:40.603021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.602991 5072 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:30:40.603021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603013 5072 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:30:40.603187 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603034 5072 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:30:40.603241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603210 5072 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 05:30:40.603277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603224 5072 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 05:30:40.603277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603254 5072 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 05:30:40.603277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603262 5072 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 05:30:40.603277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603270 5072 policy_none.go:50] "Start" Oct 30 05:30:40.603386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603303 5072 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:30:40.603386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603317 5072 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:30:40.603575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603563 5072 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 05:30:40.603607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603578 5072 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 05:30:40.603607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.603587 5072 policy_none.go:44] "Start" Oct 30 05:30:40.608447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.608407 5072 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:30:40.610361 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:40.610337 5072 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 05:30:40.611039 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.611027 5072 manager.go:341] "Starting Device Plugin manager" Oct 30 05:30:40.611584 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:40.611052 5072 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:30:40.611584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.611586 5072 server.go:87] "Starting device plugin registration server" Oct 30 05:30:40.611838 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.611828 5072 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:30:40.611920 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.611840 5072 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:30:40.611983 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.611952 5072 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:30:40.612097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.612059 5072 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:30:40.612292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.612106 5072 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:30:40.612292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.612114 5072 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:30:40.613099 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.612182 5072 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:30:40.612095203Z","UUID":"821659b4-bdc7-4dd2-a828-36a2f62e0812","Handler":null,"Name":"","Endpoint":""} Oct 30 05:30:40.613671 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:40.613403 5072 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:30:40.613671 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.613557 5072 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:30:40.613671 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.613590 5072 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:30:40.712989 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.712964 5072 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:30:40.714133 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.714118 5072 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:30:40.714201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.714147 5072 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:30:40.714201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.714157 5072 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:30:40.714201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.714201 5072 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.749108 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.749083 5072 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.749212 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.749144 5072 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.810940 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.810860 5072 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"] Oct 30 05:30:40.811509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.811451 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" event={"ID":"fd03e27bf6704eb151f6eb79c368f447","Type":"ContainerStarted","Data":"f212a1d11ee683823ab0c862bcd91e9cd98b5bb668da4dc1104aa7cae122840f"} Oct 30 05:30:40.811645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.811513 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" event={"ID":"fd03e27bf6704eb151f6eb79c368f447","Type":"ContainerStarted","Data":"6e680adc9c3cba9823577b254fd10ef66223c00ece6a78ce082dcb9d97b39d93"} Oct 30 05:30:40.811645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.811527 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerStarted","Data":"3dadcb6da8af20a47e583500a8ea6985c62b0bbf2f3dcd0b99584c8df7e58fc7"} Oct 30 05:30:40.811645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.811539 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerDied","Data":"d24f14bcbd1e4d496f361468e715510fe2777a9870a4cb50df5103ac557bf6ec"} Oct 30 05:30:40.811645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.811553 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerStarted","Data":"dfc163cc70ad703354b5534c2a01765a1b84d0b36b3faa95673a4e97ab17619e"} Oct 30 05:30:40.811645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.811586 5072 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.811910 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.811702 5072 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.838386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.838362 5072 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:30:40.838530 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:40.838409 5072 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.839025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.839010 5072 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:30:40.839064 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:40.839046 5072 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.926386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.926352 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.926386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.926384 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:40.926569 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:40.926406 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.026996 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.026963 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.027142 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.027051 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.027142 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.027059 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.027142 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.027068 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.027142 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.027095 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.027142 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.027129 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.459549 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.459509 5072 apiserver.go:52] "Watching apiserver" Oct 30 05:30:41.521250 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.521222 5072 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:30:41.521991 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.521964 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj","openshift-network-console/networking-console-plugin-86c478bb88-l725b","openshift-multus/multus-jlq5d","openshift-ingress/router-default-54dbcc76d9-x7c5h","openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d","openshift-multus/network-metrics-daemon-mtw44","openshift-image-registry/image-registry-869865c857-td99r","openshift-image-registry/image-registry-cc48cb9d8-jzp2t","openshift-network-operator/iptables-alerter-78m84","kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr","kube-system/global-pull-secret-syncer-g6mhk","openshift-insights/insights-runtime-extractor-tlm2v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal","openshift-dns/node-resolver-trbn7","kube-system/kas-connection-checker-685895c7c9-kb9n4","openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv","kube-system/kas-connection-checker-685895c7c9-4fnpv","openshift-network-diagnostics/network-check-target-bmv92","openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn","openshift-ingress-canary/ingress-canary-x2s8x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw","openshift-insights/periodic-gathering-6hlgb-qtlr7","openshift-console-operator/console-operator-7cd56b8dbc-c9p4j","openshift-image-registry/node-ca-vltvd","openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt","openshift-service-ca/service-ca-5fffc97bdf-wgrjc","openshift-ovn-kubernetes/ovnkube-node-d2qsz","kube-system/konnectivity-agent-scqnq","openshift-multus/multus-additional-cni-plugins-kt95j","openshift-network-diagnostics/network-check-source-6f78d59f99-846hg","openshift-insights/insights-operator-7c4986d449-wr9wr","openshift-cluster-node-tuning-operator/tuned-gzrqp","kube-system/kas-connection-checker-685895c7c9-l5k87","openshift-dns/dns-default-bnfpg"] Oct 30 05:30:41.524102 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.524079 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.525711 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.525691 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:41.526732 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.526715 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:41.527181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.527162 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.528201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.528181 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:41.536678 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.536656 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:41.538706 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.538685 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.539792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.539773 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.539919 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.539805 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:41.540272 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.540251 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.544484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.541178 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 05:30:41.544484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.541176 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.544484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.541214 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 05:30:41.544484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.541213 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 05:30:41.544484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.543027 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.553469 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553440 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 05:30:41.553587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553484 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 05:30:41.553587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553497 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.553587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553517 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hc6rm\"" Oct 30 05:30:41.553587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553574 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 05:30:41.553818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553578 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 05:30:41.553818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553594 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 05:30:41.553818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553629 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.553818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553659 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.554180 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.554156 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 05:30:41.554283 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.553664 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2m9cf\"" Oct 30 05:30:41.554390 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.554361 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r79mb\"" Oct 30 05:30:41.554524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.554451 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w4h88\"" Oct 30 05:30:41.557695 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.554736 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 05:30:41.557695 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.554874 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 05:30:41.557695 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.554991 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 05:30:41.558293 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558213 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.558441 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558425 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:30:41.558547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558472 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:30:41.558547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558482 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.558547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558481 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:30:41.558687 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558212 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.558809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558793 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.558864 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558817 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.558864 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558852 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.558964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558794 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.558964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558870 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.558964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558879 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.558964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558819 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.558964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558918 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:30:41.558964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558936 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.558964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558941 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:30:41.558964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558945 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.558964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558956 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.559277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558990 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.559277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559012 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:30:41.559277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559031 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 05:30:41.559277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559015 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.559277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559038 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 05:30:41.559277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.558870 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.559757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559739 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.559848 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559747 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.559848 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559770 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-49l47\"" Oct 30 05:30:41.559848 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559748 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.559848 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559795 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5cj6p\"" Oct 30 05:30:41.559848 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559801 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.559848 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559813 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-76x48\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559867 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559886 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559896 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt4k5\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559921 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559931 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559963 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9q7xl\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559967 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.559999 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560038 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 05:30:41.560050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560053 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.560366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560058 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w87jr\"" Oct 30 05:30:41.560366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560101 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-w2lf4\"" Oct 30 05:30:41.560366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560158 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.560366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560188 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.560366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560209 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.560366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560216 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:30:41.560366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560287 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:30:41.560366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560311 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.560366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560347 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:30:41.560679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560375 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 05:30:41.560679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560427 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.560679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560476 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.560679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560511 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:30:41.560679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560543 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 05:30:41.560679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560573 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 05:30:41.560679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560619 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vgfqj\"" Oct 30 05:30:41.560679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560661 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.560946 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560691 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 05:30:41.560946 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560719 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-n79jw\"" Oct 30 05:30:41.560946 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560755 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.560946 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560790 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:30:41.560946 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.560832 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:30:41.561214 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.561192 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wnglw\"" Oct 30 05:30:41.561298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.561223 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:30:41.561298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.561234 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-897vl\"" Oct 30 05:30:41.561298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.561227 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nnkr\"" Oct 30 05:30:41.561298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.561250 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:30:41.561298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.561255 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wzzmx\"" Oct 30 05:30:41.561298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.561269 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-twkbr\"" Oct 30 05:30:41.561298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.561269 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c4j6h\"" Oct 30 05:30:41.564415 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.564399 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c6sjw\"" Oct 30 05:30:41.564415 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.564407 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-88sr6\"" Oct 30 05:30:41.566320 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.566300 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:30:41.566400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.566321 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t86pd\"" Oct 30 05:30:41.566400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.566322 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 05:30:41.566400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.566342 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t4r8b\"" Oct 30 05:30:41.566400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.566352 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.566400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.566355 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bfw5z\"" Oct 30 05:30:41.566584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.566426 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 05:30:41.566696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.566672 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9nrzf\"" Oct 30 05:30:41.566696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.566686 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:30:41.570563 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.570546 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 05:30:41.571335 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.571315 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 05:30:41.571447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.571335 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 05:30:41.571447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.571340 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:30:41.571584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.571449 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:30:41.571584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.571483 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnf9b\"" Oct 30 05:30:41.571584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.571568 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cjds4\"" Oct 30 05:30:41.571724 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.571587 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:41.573182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.573164 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 05:30:41.573900 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.573888 5072 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.574041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.574030 5072 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.577777 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.577761 5072 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:30:41.602056 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.602035 5072 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:30:41.602155 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.602080 5072 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.602155 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.602037 5072 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:30:41.602235 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.602154 5072 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" Oct 30 05:30:41.632105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632078 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.632208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632112 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-netns\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.632208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632133 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-netd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.632208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632161 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkmt\" (UniqueName: \"kubernetes.io/projected/60d71bc3-a236-4305-baeb-df6403b52f43-kube-api-access-lnkmt\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.632362 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632208 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-log-socket\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.632362 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632234 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-config\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.632362 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632267 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-trusted-ca\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:41.632362 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632324 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.632608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632369 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.632608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632386 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgps8\" (UniqueName: \"kubernetes.io/projected/540c21f9-43da-400c-b0da-e0b01eecedf7-kube-api-access-dgps8\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:41.632608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632433 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.632608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632482 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-bin\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.632805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632616 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.632805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632690 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-multus-certs\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.632805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632699 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-config\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.632805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632720 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.632805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632754 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3d3980e-8842-4c87-a275-8bf0da706fba-hosts-file\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:41.632805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632790 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-tmp\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632842 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-slash\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632852 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-tmp\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632871 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-host\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632901 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.632999 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-systemd-units\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633035 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n589\" (UniqueName: \"kubernetes.io/projected/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-kube-api-access-5n589\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633056 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ts77\" (UniqueName: \"kubernetes.io/projected/0ee3f1bf-665e-4220-ac90-5d0b041d7077-kube-api-access-6ts77\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633077 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysconfig\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633038 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.633097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633100 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-tmp\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633138 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-kubelet\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633162 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633177 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-tmp\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633215 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfrm\" (UniqueName: \"kubernetes.io/projected/afe5578f-4885-4a55-a947-4c9bfe331614-kube-api-access-wmfrm\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633256 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1e52c123-4401-4329-8058-5479f3680281-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633292 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-trusted-ca\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633305 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kstz\" (UniqueName: \"kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633383 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633411 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633412 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.633501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633483 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-system-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633510 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633552 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1e52c123-4401-4329-8058-5479f3680281-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633552 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-k8s-cni-cncf-io\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633623 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633703 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633731 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633762 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633784 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633786 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633804 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:41.633837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633833 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633918 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633943 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-multus\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633965 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-host\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.633996 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wj77\" (UniqueName: \"kubernetes.io/projected/26ef5d7f-4915-4cee-a412-c1f1f38410bd-kube-api-access-8wj77\") pod \"network-check-source-6f78d59f99-846hg\" (UID: \"26ef5d7f-4915-4cee-a412-c1f1f38410bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634018 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634021 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-systemd\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634048 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-kubelet\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634053 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634090 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-socket-dir-parent\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634121 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7khr\" (UniqueName: \"kubernetes.io/projected/f25962bc-3b24-4f15-842f-e3d9771a1809-kube-api-access-g7khr\") pod \"kas-connection-checker-685895c7c9-kb9n4\" (UID: \"f25962bc-3b24-4f15-842f-e3d9771a1809\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" Oct 30 05:30:41.634150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634147 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634189 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4wgj\" (UniqueName: \"kubernetes.io/projected/6381b6d8-a45f-43d3-955c-504694db87da-kube-api-access-p4wgj\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634238 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-serviceca\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634264 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-lib-modules\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634289 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-etc-selinux\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634321 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634322 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634366 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddvn\" (UniqueName: \"kubernetes.io/projected/d0f89396-3754-4034-af10-b9a951d3bbb1-kube-api-access-xddvn\") pod \"volume-data-source-validator-58c9f5f46d-dggnj\" (UID: \"d0f89396-3754-4034-af10-b9a951d3bbb1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634385 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634401 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-etc-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634486 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d813b7e-2c08-4c0f-b077-2b529d90cf51-serving-cert\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634493 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-serviceca\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634502 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/429c555a-f735-446f-8577-4f7670a5d094-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:41.634532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634529 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634550 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v974p\" (UniqueName: \"kubernetes.io/projected/12fe3fee-9496-4b7b-8923-d19c585cc4be-kube-api-access-v974p\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634579 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3d3980e-8842-4c87-a275-8bf0da706fba-tmp-dir\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634655 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/429c555a-f735-446f-8577-4f7670a5d094-nginx-conf\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634666 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60d71bc3-a236-4305-baeb-df6403b52f43-serving-cert\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634685 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3d3980e-8842-4c87-a275-8bf0da706fba-tmp-dir\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634694 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovn-node-metrics-cert\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634724 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634753 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634765 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60d71bc3-a236-4305-baeb-df6403b52f43-serving-cert\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634770 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634815 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-service-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634859 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/540c21f9-43da-400c-b0da-e0b01eecedf7-cert\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634876 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovn-node-metrics-cert\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.634914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634891 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634948 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxd2m\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.634981 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635008 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-env-overrides\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635034 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-kubelet-config\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635058 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-modprobe-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635056 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635087 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d813b7e-2c08-4c0f-b077-2b529d90cf51-serving-cert\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635087 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f968039-9ce3-48ea-a1c6-c963206a596c-config\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635110 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635151 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgr9\" (UniqueName: \"kubernetes.io/projected/2f968039-9ce3-48ea-a1c6-c963206a596c-kube-api-access-bqgr9\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635210 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-ovn\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635243 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635269 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-env-overrides\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635285 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/540c21f9-43da-400c-b0da-e0b01eecedf7-cert\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635300 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-systemd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635305 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f968039-9ce3-48ea-a1c6-c963206a596c-config\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:41.635324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635332 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635325 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-conf-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635376 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkvb\" (UniqueName: \"kubernetes.io/projected/8d813b7e-2c08-4c0f-b077-2b529d90cf51-kube-api-access-5kkvb\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635433 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-var-lib-kubelet\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635497 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635505 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60d71bc3-a236-4305-baeb-df6403b52f43-service-ca-bundle\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635537 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqgp\" (UniqueName: \"kubernetes.io/projected/70464c99-1f1b-49e4-8cc9-335fa1645a98-kube-api-access-rdqgp\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635587 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f968039-9ce3-48ea-a1c6-c963206a596c-serving-cert\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635620 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-bin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635643 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95b26dcb-4282-498a-b278-4c7bb6ac11a1-crio-socket\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635668 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635694 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-run\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635722 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f1bf-665e-4220-ac90-5d0b041d7077-tmp-dir\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635752 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-default-certificate\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635779 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f968039-9ce3-48ea-a1c6-c963206a596c-serving-cert\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635784 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f1bf-665e-4220-ac90-5d0b041d7077-tmp-dir\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635783 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635786 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-cabundle\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635816 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zrt\" (UniqueName: \"kubernetes.io/projected/ae0a00ed-126f-4c35-a159-aa6ed7e2c72f-kube-api-access-h5zrt\") pod \"kas-connection-checker-685895c7c9-4fnpv\" (UID: \"ae0a00ed-126f-4c35-a159-aa6ed7e2c72f\") " pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" Oct 30 05:30:41.635844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635832 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-script-lib\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635854 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635882 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74f707f9-cd75-4d5f-bab3-3ead92d87904-agent-certs\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635909 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hcv9\" (UniqueName: \"kubernetes.io/projected/671efddc-7c80-4be5-827d-0c227985c013-kube-api-access-8hcv9\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635934 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635961 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7q66\" (UniqueName: \"kubernetes.io/projected/f3d3980e-8842-4c87-a275-8bf0da706fba-kube-api-access-l7q66\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.635964 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-default-certificate\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636001 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgk6\" (UniqueName: \"kubernetes.io/projected/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-kube-api-access-spgk6\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636019 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdk9\" (UniqueName: \"kubernetes.io/projected/62348232-2a4d-4ed2-8830-10f547a59fea-kube-api-access-bjdk9\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636056 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-system-cni-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636088 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-ovnkube-script-lib\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636088 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-registration-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636149 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-sys-fs\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636211 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjbzf\" (UniqueName: \"kubernetes.io/projected/454eb54a-074f-45e9-94cb-75356bb42e71-kube-api-access-gjbzf\") pod \"migrator-66df5d496b-kl6gv\" (UID: \"454eb54a-074f-45e9-94cb-75356bb42e71\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636219 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74f707f9-cd75-4d5f-bab3-3ead92d87904-agent-certs\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636264 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78ks\" (UniqueName: \"kubernetes.io/projected/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-kube-api-access-n78ks\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636310 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.636526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636413 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.637192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636746 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttx6t\" (UniqueName: \"kubernetes.io/projected/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-kube-api-access-ttx6t\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.637192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636800 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-iptables-alerter-script\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:41.637192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636807 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.637192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636830 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-kubernetes\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.637192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.636867 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqgpk\" (UniqueName: \"kubernetes.io/projected/1e52c123-4401-4329-8058-5479f3680281-kube-api-access-kqgpk\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:41.637192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637158 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-dbus\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:41.637505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637221 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-netns\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.637505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637248 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-iptables-alerter-script\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:41.637505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637270 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.637505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637327 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-host-slash\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:41.637505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637397 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.637505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637437 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-snapshots\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.637505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637484 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.637843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637521 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:41.637843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637571 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-config\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:41.637843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637580 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60d71bc3-a236-4305-baeb-df6403b52f43-snapshots\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.637843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637777 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d813b7e-2c08-4c0f-b077-2b529d90cf51-config\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:41.637843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637825 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee3f1bf-665e-4220-ac90-5d0b041d7077-config-volume\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:41.638074 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637879 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-stats-auth\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.638074 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637909 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-original-pull-secret\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:41.638074 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.637952 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.638224 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638088 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.638224 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638143 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-cnibin\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.638224 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638172 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.638368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638251 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.638368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638287 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-socket-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.638368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638325 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzgj\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.638526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638360 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.638526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638435 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-daemon-config\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.638526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638506 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-sys\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.638677 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638545 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:41.638677 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638579 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-var-lib-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.638677 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638615 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:41.638677 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638645 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-etc-tuned\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.638859 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638679 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-device-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.638859 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638819 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/afe5578f-4885-4a55-a947-4c9bfe331614-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:41.638953 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638857 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-node-log\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.638953 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638894 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.638953 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638927 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74f707f9-cd75-4d5f-bab3-3ead92d87904-konnectivity-ca\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:41.639095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638959 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-conf\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.639095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.638997 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48wsv\" (UniqueName: \"kubernetes.io/projected/510573e7-a380-415f-9609-18470c856703-kube-api-access-48wsv\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.639095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639030 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-cni-binary-copy\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.639095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639057 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-hostroot\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.639095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639089 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-etc-kubernetes\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.639334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639124 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhq8c\" (UniqueName: \"kubernetes.io/projected/af48072f-2e02-46db-9b2d-7eb2da4cd2e4-kube-api-access-nhq8c\") pod \"migrator-66df5d496b-5z7dt\" (UID: \"af48072f-2e02-46db-9b2d-7eb2da4cd2e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" Oct 30 05:30:41.639334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639157 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-key\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:41.639334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639193 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjx5n\" (UniqueName: \"kubernetes.io/projected/9e2abeba-b181-40af-89c2-c3782ad160c5-kube-api-access-vjx5n\") pod \"kas-connection-checker-685895c7c9-l5k87\" (UID: \"9e2abeba-b181-40af-89c2-c3782ad160c5\") " pod="kube-system/kas-connection-checker-685895c7c9-l5k87" Oct 30 05:30:41.639334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639223 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6f78\" (UniqueName: \"kubernetes.io/projected/e0a3b194-c453-404f-92a4-0b1ab87f7699-kube-api-access-v6f78\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.639334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639275 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b26dcb-4282-498a-b278-4c7bb6ac11a1-data-volume\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.639334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639316 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7gk\" (UniqueName: \"kubernetes.io/projected/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-api-access-pt7gk\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.641450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639734 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-cnibin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.641450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.639783 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.641450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.640057 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/510573e7-a380-415f-9609-18470c856703-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.641450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.640590 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-stats-auth\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.641450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.640753 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee3f1bf-665e-4220-ac90-5d0b041d7077-config-volume\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:41.641450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.641333 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74f707f9-cd75-4d5f-bab3-3ead92d87904-konnectivity-ca\") pod \"konnectivity-agent-scqnq\" (UID: \"74f707f9-cd75-4d5f-bab3-3ead92d87904\") " pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:41.641704 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.641513 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62348232-2a4d-4ed2-8830-10f547a59fea-etc-tuned\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.641741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.641701 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/afe5578f-4885-4a55-a947-4c9bfe331614-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:41.641775 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.641747 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:41.641775 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.641765 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95b26dcb-4282-498a-b278-4c7bb6ac11a1-data-volume\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.641943 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.641925 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-daemon-config\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.642362 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.642341 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/70464c99-1f1b-49e4-8cc9-335fa1645a98-signing-key\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:41.642406 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.642370 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a3b194-c453-404f-92a4-0b1ab87f7699-cni-binary-copy\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.653612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.653589 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnkmt\" (UniqueName: \"kubernetes.io/projected/60d71bc3-a236-4305-baeb-df6403b52f43-kube-api-access-lnkmt\") pod \"insights-operator-7c4986d449-wr9wr\" (UID: \"60d71bc3-a236-4305-baeb-df6403b52f43\") " pod="openshift-insights/insights-operator-7c4986d449-wr9wr" Oct 30 05:30:41.657016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.657000 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgps8\" (UniqueName: \"kubernetes.io/projected/540c21f9-43da-400c-b0da-e0b01eecedf7-kube-api-access-dgps8\") pod \"ingress-canary-x2s8x\" (UID: \"540c21f9-43da-400c-b0da-e0b01eecedf7\") " pod="openshift-ingress-canary/ingress-canary-x2s8x" Oct 30 05:30:41.659260 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.659237 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfrm\" (UniqueName: \"kubernetes.io/projected/afe5578f-4885-4a55-a947-4c9bfe331614-kube-api-access-wmfrm\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:41.659425 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.659408 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n589\" (UniqueName: \"kubernetes.io/projected/4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d-kube-api-access-5n589\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jptrr\" (UID: \"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" Oct 30 05:30:41.659762 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.659744 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ts77\" (UniqueName: \"kubernetes.io/projected/0ee3f1bf-665e-4220-ac90-5d0b041d7077-kube-api-access-6ts77\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:41.659971 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.659955 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.660088 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.660066 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgr9\" (UniqueName: \"kubernetes.io/projected/2f968039-9ce3-48ea-a1c6-c963206a596c-kube-api-access-bqgr9\") pod \"service-ca-operator-544d749c8c-f2stn\" (UID: \"2f968039-9ce3-48ea-a1c6-c963206a596c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" Oct 30 05:30:41.660126 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.660085 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kstz\" (UniqueName: \"kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz\") pod \"periodic-gathering-6hlgb-qtlr7\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:30:41.660215 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.660200 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hcv9\" (UniqueName: \"kubernetes.io/projected/671efddc-7c80-4be5-827d-0c227985c013-kube-api-access-8hcv9\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:41.668488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.668432 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.668624 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.668503 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgk6\" (UniqueName: \"kubernetes.io/projected/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-kube-api-access-spgk6\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:41.668624 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.668587 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxd2m\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.668779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.668690 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqgp\" (UniqueName: \"kubernetes.io/projected/70464c99-1f1b-49e4-8cc9-335fa1645a98-kube-api-access-rdqgp\") pod \"service-ca-5fffc97bdf-wgrjc\" (UID: \"70464c99-1f1b-49e4-8cc9-335fa1645a98\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wgrjc" Oct 30 05:30:41.668779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.668728 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkvb\" (UniqueName: \"kubernetes.io/projected/8d813b7e-2c08-4c0f-b077-2b529d90cf51-kube-api-access-5kkvb\") pod \"console-operator-7cd56b8dbc-c9p4j\" (UID: \"8d813b7e-2c08-4c0f-b077-2b529d90cf51\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:41.668779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.668742 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddvn\" (UniqueName: \"kubernetes.io/projected/d0f89396-3754-4034-af10-b9a951d3bbb1-kube-api-access-xddvn\") pod \"volume-data-source-validator-58c9f5f46d-dggnj\" (UID: \"d0f89396-3754-4034-af10-b9a951d3bbb1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dggnj" Oct 30 05:30:41.668958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.668812 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wj77\" (UniqueName: \"kubernetes.io/projected/26ef5d7f-4915-4cee-a412-c1f1f38410bd-kube-api-access-8wj77\") pod \"network-check-source-6f78d59f99-846hg\" (UID: \"26ef5d7f-4915-4cee-a412-c1f1f38410bd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-846hg" Oct 30 05:30:41.668958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.668909 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v974p\" (UniqueName: \"kubernetes.io/projected/12fe3fee-9496-4b7b-8923-d19c585cc4be-kube-api-access-v974p\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.669061 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.668967 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjbzf\" (UniqueName: \"kubernetes.io/projected/454eb54a-074f-45e9-94cb-75356bb42e71-kube-api-access-gjbzf\") pod \"migrator-66df5d496b-kl6gv\" (UID: \"454eb54a-074f-45e9-94cb-75356bb42e71\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-kl6gv" Oct 30 05:30:41.669061 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.669037 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78ks\" (UniqueName: \"kubernetes.io/projected/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-kube-api-access-n78ks\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:41.669173 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.669130 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zrt\" (UniqueName: \"kubernetes.io/projected/ae0a00ed-126f-4c35-a159-aa6ed7e2c72f-kube-api-access-h5zrt\") pod \"kas-connection-checker-685895c7c9-4fnpv\" (UID: \"ae0a00ed-126f-4c35-a159-aa6ed7e2c72f\") " pod="kube-system/kas-connection-checker-685895c7c9-4fnpv" Oct 30 05:30:41.669310 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.669291 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7khr\" (UniqueName: \"kubernetes.io/projected/f25962bc-3b24-4f15-842f-e3d9771a1809-kube-api-access-g7khr\") pod \"kas-connection-checker-685895c7c9-kb9n4\" (UID: \"f25962bc-3b24-4f15-842f-e3d9771a1809\") " pod="kube-system/kas-connection-checker-685895c7c9-kb9n4" Oct 30 05:30:41.670332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.670313 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqgpk\" (UniqueName: \"kubernetes.io/projected/1e52c123-4401-4329-8058-5479f3680281-kube-api-access-kqgpk\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:41.670656 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.670637 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjx5n\" (UniqueName: \"kubernetes.io/projected/9e2abeba-b181-40af-89c2-c3782ad160c5-kube-api-access-vjx5n\") pod \"kas-connection-checker-685895c7c9-l5k87\" (UID: \"9e2abeba-b181-40af-89c2-c3782ad160c5\") " pod="kube-system/kas-connection-checker-685895c7c9-l5k87" Oct 30 05:30:41.670809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.670777 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttx6t\" (UniqueName: \"kubernetes.io/projected/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-kube-api-access-ttx6t\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.670950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.670933 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzgj\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.671507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.671490 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7gk\" (UniqueName: \"kubernetes.io/projected/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-api-access-pt7gk\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.671573 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.671493 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48wsv\" (UniqueName: \"kubernetes.io/projected/510573e7-a380-415f-9609-18470c856703-kube-api-access-48wsv\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.672645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.672626 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdk9\" (UniqueName: \"kubernetes.io/projected/62348232-2a4d-4ed2-8830-10f547a59fea-kube-api-access-bjdk9\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.674379 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.674362 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c926\" (UniqueName: \"kubernetes.io/projected/0d304ba5-b889-412c-ae7d-b26f04222714-kube-api-access-5c926\") pod \"network-check-target-bmv92\" (UID: \"0d304ba5-b889-412c-ae7d-b26f04222714\") " pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:41.675229 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.675210 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7q66\" (UniqueName: \"kubernetes.io/projected/f3d3980e-8842-4c87-a275-8bf0da706fba-kube-api-access-l7q66\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:41.675568 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.675550 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4wgj\" (UniqueName: \"kubernetes.io/projected/6381b6d8-a45f-43d3-955c-504694db87da-kube-api-access-p4wgj\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.677292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.677276 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6f78\" (UniqueName: \"kubernetes.io/projected/e0a3b194-c453-404f-92a4-0b1ab87f7699-kube-api-access-v6f78\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.677566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.677551 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhq8c\" (UniqueName: \"kubernetes.io/projected/af48072f-2e02-46db-9b2d-7eb2da4cd2e4-kube-api-access-nhq8c\") pod \"migrator-66df5d496b-5z7dt\" (UID: \"af48072f-2e02-46db-9b2d-7eb2da4cd2e4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5z7dt" Oct 30 05:30:41.740884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.740824 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-system-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.740884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.740851 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-k8s-cni-cncf-io\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.740884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.740868 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.740884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.740883 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:41.741134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.740939 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-k8s-cni-cncf-io\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.741134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.740956 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-system-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.741134 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.740985 5072 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:30:41.741134 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.741062 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:42.241048059 +0000 UTC m=+1.844737312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : secret "insights-runtime-extractor-tls" not found Oct 30 05:30:41.741134 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.741097 5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:41.741134 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.741114 5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc48cb9d8-jzp2t: secret "image-registry-tls" not found Oct 30 05:30:41.741134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741131 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741152 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-multus\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.741181 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls podName:35863a55-5422-4304-ad71-afddab4d4807 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:42.241162396 +0000 UTC m=+1.844851653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls") pod "image-registry-cc48cb9d8-jzp2t" (UID: "35863a55-5422-4304-ad71-afddab4d4807") : secret "image-registry-tls" not found Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741187 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-multus\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741200 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-host\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741213 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741233 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37-host\") pod \"node-ca-vltvd\" (UID: \"3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37\") " pod="openshift-image-registry/node-ca-vltvd" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741241 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-systemd\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741266 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-kubelet\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741296 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-socket-dir-parent\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741323 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-lib-modules\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741347 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-systemd\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741351 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-kubelet\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741371 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-socket-dir-parent\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741383 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-etc-selinux\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741420 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741449 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741478 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-lib-modules\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741483 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-etc-selinux\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.741477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741488 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-etc-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741521 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741531 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-etc-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741544 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741563 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741579 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-kubelet-config\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741602 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-modprobe-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741630 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-cni-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741637 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741627 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-ovn\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741672 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-systemd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741684 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-kubelet-config\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741694 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-conf-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741714 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-var-lib-kubelet\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741723 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-ovn\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741736 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-bin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741743 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-run-systemd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741759 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-multus-conf-dir\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741760 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95b26dcb-4282-498a-b278-4c7bb6ac11a1-crio-socket\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741776 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-var-lib-kubelet\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741771 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-modprobe-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741797 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95b26dcb-4282-498a-b278-4c7bb6ac11a1-crio-socket\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741794 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-cni-bin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741797 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741782 5072 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741889 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-run\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741908 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741927 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-d\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741967 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-run\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.741934 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.742003 5072 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.742008 5072 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742040 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-system-cni-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.742065 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls podName:1e52c123-4401-4329-8058-5479f3680281 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:42.242046899 +0000 UTC m=+1.845736153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gx2pw" (UID: "1e52c123-4401-4329-8058-5479f3680281") : secret "samples-operator-tls" not found Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742010 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-system-cni-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.742089 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls podName:0ee3f1bf-665e-4220-ac90-5d0b041d7077 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:42.242074713 +0000 UTC m=+1.845763954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls") pod "dns-default-bnfpg" (UID: "0ee3f1bf-665e-4220-ac90-5d0b041d7077") : secret "dns-default-metrics-tls" not found Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742112 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-registration-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742152 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-sys-fs\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742187 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-kubernetes\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742209 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-dbus\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742228 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-registration-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742230 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-netns\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742265 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-netns\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742316 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde-dbus\") pod \"global-pull-secret-syncer-g6mhk\" (UID: \"56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde\") " pod="kube-system/global-pull-secret-syncer-g6mhk" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742361 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-kubernetes\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742264 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742392 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-host-slash\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742449 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9db8ada4-0aec-4b8d-844b-e7d9abbe0850-host-slash\") pod \"iptables-alerter-78m84\" (UID: \"9db8ada4-0aec-4b8d-844b-e7d9abbe0850\") " pod="openshift-network-operator/iptables-alerter-78m84" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742521 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-cnibin\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.742523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742548 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742572 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-socket-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742597 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742623 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-sys\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742646 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-var-lib-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742673 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-device-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742697 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-node-log\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742718 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742744 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-conf\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742770 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-hostroot\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742792 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-etc-kubernetes\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742820 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-cnibin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742848 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-netns\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742861 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-var-lib-openvswitch\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742910 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-etc-kubernetes\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742918 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-netd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742957 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-netns\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742962 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-device-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742999 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-cnibin\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743005 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-node-log\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743046 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742965 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-hostroot\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743087 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysctl-conf\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743089 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-socket-dir\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.742872 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-netd\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743124 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-log-socket\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743127 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-cnibin\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743129 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743150 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-sys\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743091 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6381b6d8-a45f-43d3-955c-504694db87da-service-ca-bundle\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743180 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743236 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-bin\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743203 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-log-socket\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743247 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/510573e7-a380-415f-9609-18470c856703-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt95j\" (UID: \"510573e7-a380-415f-9609-18470c856703\") " pod="openshift-multus/multus-additional-cni-plugins-kt95j" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743259 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-multus-certs\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743283 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3d3980e-8842-4c87-a275-8bf0da706fba-hosts-file\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743286 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-cni-bin\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743305 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-slash\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743325 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-run-multus-certs\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743329 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-host\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743355 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-systemd-units\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743366 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3d3980e-8842-4c87-a275-8bf0da706fba-hosts-file\") pod \"node-resolver-trbn7\" (UID: \"f3d3980e-8842-4c87-a275-8bf0da706fba\") " pod="openshift-dns/node-resolver-trbn7" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743380 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysconfig\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743404 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-kubelet\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743435 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-host-slash\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743443 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743499 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743505 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3-systemd-units\") pod \"ovnkube-node-d2qsz\" (UID: \"cfc9f3ff-b5e8-4031-8a21-301c46e1bcf3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743404 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-host\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743587 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62348232-2a4d-4ed2-8830-10f547a59fea-etc-sysconfig\") pod \"tuned-gzrqp\" (UID: \"62348232-2a4d-4ed2-8830-10f547a59fea\") " pod="openshift-cluster-node-tuning-operator/tuned-gzrqp" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.743607 5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.743620 5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869865c857-td99r: secret "image-registry-tls" not found Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.743616 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0a3b194-c453-404f-92a4-0b1ab87f7699-host-var-lib-kubelet\") pod \"multus-jlq5d\" (UID: \"e0a3b194-c453-404f-92a4-0b1ab87f7699\") " pod="openshift-multus/multus-jlq5d" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:41.743677 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls podName:2fd827f0-3153-4b94-8326-8c6fd0744f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:42.243660996 +0000 UTC m=+1.847350236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls") pod "image-registry-869865c857-td99r" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b") : secret "image-registry-tls" not found Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.744005 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95b26dcb-4282-498a-b278-4c7bb6ac11a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.744052 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12fe3fee-9496-4b7b-8923-d19c585cc4be-sys-fs\") pod \"aws-ebs-csi-driver-node-nl8t8\" (UID: \"12fe3fee-9496-4b7b-8923-d19c585cc4be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl8t8" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.744693 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/429c555a-f735-446f-8577-4f7670a5d094-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-l725b\" (UID: \"429c555a-f735-446f-8577-4f7670a5d094\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:41.745011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.744767 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/afe5578f-4885-4a55-a947-4c9bfe331614-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4mn4d\" (UID: \"afe5578f-4885-4a55-a947-4c9bfe331614\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:41.746824 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.745248 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/671efddc-7c80-4be5-827d-0c227985c013-metrics-certs\") pod \"network-metrics-daemon-mtw44\" (UID: \"671efddc-7c80-4be5-827d-0c227985c013\") " pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:41.746824 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.746007 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6381b6d8-a45f-43d3-955c-504694db87da-metrics-certs\") pod \"router-default-54dbcc76d9-x7c5h\" (UID: \"6381b6d8-a45f-43d3-955c-504694db87da\") " pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.825654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.825625 5072 scope.go:122] "RemoveContainer" containerID="76737ca3daa3bc1b7826bbb28c96768c103e4eb44fcfedc551d1ee20fc521585" Oct 30 05:30:41.825940 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.825923 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" Oct 30 05:30:41.827526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.827496 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:41.828540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.828524 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" Oct 30 05:30:41.836916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:41.836895 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtw44" Oct 30 05:30:42.018723 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.018650 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vltvd_3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37/node-ca/0.log" Oct 30 05:30:42.143488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.143446 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d"] Oct 30 05:30:42.145479 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:42.145429 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe5578f_4885_4a55_a947_4c9bfe331614.slice/crio-a6370889690c441cf0230258689162be465ff8257586841fe74e77f3758ee670 WatchSource:0}: Error finding container a6370889690c441cf0230258689162be465ff8257586841fe74e77f3758ee670: Status 404 returned error can't find the container with id a6370889690c441cf0230258689162be465ff8257586841fe74e77f3758ee670 Oct 30 05:30:42.147641 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.147623 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:30:42.171040 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.171022 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-l725b"] Oct 30 05:30:42.173139 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:42.173117 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429c555a_f735_446f_8577_4f7670a5d094.slice/crio-c77b278aa34c7b86b2b50bb735ed320374a715248ead76320aba0be1eb9ab4e3 WatchSource:0}: Error finding container c77b278aa34c7b86b2b50bb735ed320374a715248ead76320aba0be1eb9ab4e3: Status 404 returned error can't find the container with id c77b278aa34c7b86b2b50bb735ed320374a715248ead76320aba0be1eb9ab4e3 Oct 30 05:30:42.183322 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.183284 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54dbcc76d9-x7c5h"] Oct 30 05:30:42.185468 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.185435 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mtw44"] Oct 30 05:30:42.185896 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:42.185871 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6381b6d8_a45f_43d3_955c_504694db87da.slice/crio-ead3bbff31f9d7a8e8696023c6dac2be301c08ac636cbe77f473440d77a25a19 WatchSource:0}: Error finding container ead3bbff31f9d7a8e8696023c6dac2be301c08ac636cbe77f473440d77a25a19: Status 404 returned error can't find the container with id ead3bbff31f9d7a8e8696023c6dac2be301c08ac636cbe77f473440d77a25a19 Oct 30 05:30:42.188325 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:42.188302 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671efddc_7c80_4be5_827d_0c227985c013.slice/crio-a58f8a83e8b26b66c243639c133f931389c520cfab2b4921706c9edc02460fbd WatchSource:0}: Error finding container a58f8a83e8b26b66c243639c133f931389c520cfab2b4921706c9edc02460fbd: Status 404 returned error can't find the container with id a58f8a83e8b26b66c243639c133f931389c520cfab2b4921706c9edc02460fbd Oct 30 05:30:42.248575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.248549 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:42.248713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.248586 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:42.248713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.248606 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:42.248713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.248654 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:42.248713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.248681 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248731 5072 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248800 5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248808 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:43.248788764 +0000 UTC m=+2.852478015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : secret "insights-runtime-extractor-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248814 5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc48cb9d8-jzp2t: secret "image-registry-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248735 5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248830 5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869865c857-td99r: secret "image-registry-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248851 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls podName:35863a55-5422-4304-ad71-afddab4d4807 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:43.248839589 +0000 UTC m=+2.852528839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls") pod "image-registry-cc48cb9d8-jzp2t" (UID: "35863a55-5422-4304-ad71-afddab4d4807") : secret "image-registry-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248859 5072 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248866 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls podName:2fd827f0-3153-4b94-8326-8c6fd0744f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:43.248857992 +0000 UTC m=+2.852547239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls") pod "image-registry-869865c857-td99r" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b") : secret "image-registry-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248906 5072 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248906 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls podName:1e52c123-4401-4329-8058-5479f3680281 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:43.24889248 +0000 UTC m=+2.852581717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gx2pw" (UID: "1e52c123-4401-4329-8058-5479f3680281") : secret "samples-operator-tls" not found Oct 30 05:30:42.248926 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:42.248935 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls podName:0ee3f1bf-665e-4220-ac90-5d0b041d7077 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:43.248928616 +0000 UTC m=+2.852617851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls") pod "dns-default-bnfpg" (UID: "0ee3f1bf-665e-4220-ac90-5d0b041d7077") : secret "dns-default-metrics-tls" not found Oct 30 05:30:42.419416 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.419382 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x2s8x_540c21f9-43da-400c-b0da-e0b01eecedf7/serve-healthcheck-canary/0.log" Oct 30 05:30:42.580624 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.580586 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" event={"ID":"6381b6d8-a45f-43d3-955c-504694db87da","Type":"ContainerStarted","Data":"ba0de8332ebd69b5633e8039f9d25c8f7337cd50f6590a6d8a91cb7f793380a7"} Oct 30 05:30:42.581650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.580636 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" event={"ID":"6381b6d8-a45f-43d3-955c-504694db87da","Type":"ContainerStarted","Data":"ead3bbff31f9d7a8e8696023c6dac2be301c08ac636cbe77f473440d77a25a19"} Oct 30 05:30:42.582762 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.582719 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtw44" event={"ID":"671efddc-7c80-4be5-827d-0c227985c013","Type":"ContainerStarted","Data":"a58f8a83e8b26b66c243639c133f931389c520cfab2b4921706c9edc02460fbd"} Oct 30 05:30:42.584564 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.584531 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" event={"ID":"afe5578f-4885-4a55-a947-4c9bfe331614","Type":"ContainerStarted","Data":"a6370889690c441cf0230258689162be465ff8257586841fe74e77f3758ee670"} Oct 30 05:30:42.586168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.586144 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" event={"ID":"429c555a-f735-446f-8577-4f7670a5d094","Type":"ContainerStarted","Data":"c77b278aa34c7b86b2b50bb735ed320374a715248ead76320aba0be1eb9ab4e3"} Oct 30 05:30:42.589413 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.589392 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:30:42.589545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.589481 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" event={"ID":"8d813b7e-2c08-4c0f-b077-2b529d90cf51","Type":"ContainerStarted","Data":"9ff199ba69e908fb17bda92ce65f5a10526f2fd5b808c99064c2703bf07c9f01"} Oct 30 05:30:42.633001 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.632929 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5z7dt_af48072f-2e02-46db-9b2d-7eb2da4cd2e4/migrator/0.log" Oct 30 05:30:42.821579 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.821485 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-kl6gv_454eb54a-074f-45e9-94cb-75356bb42e71/migrator/0.log" Oct 30 05:30:42.828072 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.828045 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:42.831165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:42.831141 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:43.024335 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.024303 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-jptrr_4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d/kube-storage-version-migrator-operator/0.log" Oct 30 05:30:43.257591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.257566 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:43.257729 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.257611 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:43.257729 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.257644 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:43.257729 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.257707 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:43.257886 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257731 5072 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:30:43.257886 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257802 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:45.257784448 +0000 UTC m=+4.861473701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : secret "insights-runtime-extractor-tls" not found Oct 30 05:30:43.257886 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257821 5072 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:30:43.257886 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257825 5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:43.257886 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257846 5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc48cb9d8-jzp2t: secret "image-registry-tls" not found Oct 30 05:30:43.257886 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257875 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls podName:1e52c123-4401-4329-8058-5479f3680281 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:45.257859232 +0000 UTC m=+4.861548481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gx2pw" (UID: "1e52c123-4401-4329-8058-5479f3680281") : secret "samples-operator-tls" not found Oct 30 05:30:43.257886 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257734 5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:43.258186 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257894 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls podName:35863a55-5422-4304-ad71-afddab4d4807 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:45.257881566 +0000 UTC m=+4.861570817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls") pod "image-registry-cc48cb9d8-jzp2t" (UID: "35863a55-5422-4304-ad71-afddab4d4807") : secret "image-registry-tls" not found Oct 30 05:30:43.258186 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257895 5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869865c857-td99r: secret "image-registry-tls" not found Oct 30 05:30:43.258186 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.257733 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:43.258186 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:43.257922 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls podName:2fd827f0-3153-4b94-8326-8c6fd0744f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:45.257916097 +0000 UTC m=+4.861605359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls") pod "image-registry-869865c857-td99r" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b") : secret "image-registry-tls" not found Oct 30 05:30:43.260178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.260158 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee3f1bf-665e-4220-ac90-5d0b041d7077-metrics-tls\") pod \"dns-default-bnfpg\" (UID: \"0ee3f1bf-665e-4220-ac90-5d0b041d7077\") " pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:43.340055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.340033 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:43.596751 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.596666 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtw44" event={"ID":"671efddc-7c80-4be5-827d-0c227985c013","Type":"ContainerStarted","Data":"989b5c1fa9d535cfc3d4d59b51e3f2efc7e5e74460cbf5d9b621e731a9ea450f"} Oct 30 05:30:43.596751 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.596732 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtw44" event={"ID":"671efddc-7c80-4be5-827d-0c227985c013","Type":"ContainerStarted","Data":"33dbaf68c9721ae2b884862916839f45fba63c0178bcaac547f9c2d51dda95dc"} Oct 30 05:30:43.598664 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.598636 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-l725b" event={"ID":"429c555a-f735-446f-8577-4f7670a5d094","Type":"ContainerStarted","Data":"90660e6fcce304c54efc17b3c8df72fed4e333ec4e0ce1b679ce921e00f266a4"} Oct 30 05:30:43.658520 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:43.658498 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bnfpg"] Oct 30 05:30:43.660784 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:43.660745 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee3f1bf_665e_4220_ac90_5d0b041d7077.slice/crio-86a76d5273552619e52232a74893dd1debb953a83ca43eb6936757b99227be7b WatchSource:0}: Error finding container 86a76d5273552619e52232a74893dd1debb953a83ca43eb6936757b99227be7b: Status 404 returned error can't find the container with id 86a76d5273552619e52232a74893dd1debb953a83ca43eb6936757b99227be7b Oct 30 05:30:44.508936 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:44.508856 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-scqnq" Oct 30 05:30:44.605687 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:44.605647 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4mn4d" event={"ID":"afe5578f-4885-4a55-a947-4c9bfe331614","Type":"ContainerStarted","Data":"767907af5637fe140be9bd50b23511e4ac13df00d40e4c01b7d1cb10c864322b"} Oct 30 05:30:44.607408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:44.607380 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bnfpg" event={"ID":"0ee3f1bf-665e-4220-ac90-5d0b041d7077","Type":"ContainerStarted","Data":"86a76d5273552619e52232a74893dd1debb953a83ca43eb6936757b99227be7b"} Oct 30 05:30:44.607665 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:44.607649 5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:30:45.279936 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.279897 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:45.280089 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.279944 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:45.280089 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280028 5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:45.280089 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280044 5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869865c857-td99r: secret "image-registry-tls" not found Oct 30 05:30:45.280089 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280033 5072 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 05:30:45.280089 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280092 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls podName:2fd827f0-3153-4b94-8326-8c6fd0744f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:49.280074645 +0000 UTC m=+8.883763879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls") pod "image-registry-869865c857-td99r" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b") : secret "image-registry-tls" not found Oct 30 05:30:45.280396 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280116 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls podName:95b26dcb-4282-498a-b278-4c7bb6ac11a1 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:49.280099193 +0000 UTC m=+8.883788449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlm2v" (UID: "95b26dcb-4282-498a-b278-4c7bb6ac11a1") : secret "insights-runtime-extractor-tls" not found Oct 30 05:30:45.280396 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.280139 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:45.280396 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.280220 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:45.280396 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280286 5072 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 05:30:45.280396 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280301 5072 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc48cb9d8-jzp2t: secret "image-registry-tls" not found Oct 30 05:30:45.280396 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280325 5072 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 05:30:45.280396 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280342 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls podName:35863a55-5422-4304-ad71-afddab4d4807 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:49.280327583 +0000 UTC m=+8.884016819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls") pod "image-registry-cc48cb9d8-jzp2t" (UID: "35863a55-5422-4304-ad71-afddab4d4807") : secret "image-registry-tls" not found Oct 30 05:30:45.280396 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:30:45.280358 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls podName:1e52c123-4401-4329-8058-5479f3680281 nodeName:}" failed. No retries permitted until 2026-10-30 05:30:49.280349759 +0000 UTC m=+8.884039025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gx2pw" (UID: "1e52c123-4401-4329-8058-5479f3680281") : secret "samples-operator-tls" not found Oct 30 05:30:45.613159 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.613125 5072 generic.go:356] "Generic (PLEG): container finished" podID="60d71bc3-a236-4305-baeb-df6403b52f43" containerID="f6bcda01cf71459069982c449fa9598488ff34e98074ca971842a59dba058767" exitCode=0 Oct 30 05:30:45.613592 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.613196 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" event={"ID":"60d71bc3-a236-4305-baeb-df6403b52f43","Type":"ContainerDied","Data":"f6bcda01cf71459069982c449fa9598488ff34e98074ca971842a59dba058767"} Oct 30 05:30:45.613592 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.613575 5072 scope.go:122] "RemoveContainer" containerID="f6bcda01cf71459069982c449fa9598488ff34e98074ca971842a59dba058767" Oct 30 05:30:45.615270 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.615251 5072 generic.go:356] "Generic (PLEG): container finished" podID="4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d" containerID="0de7de9f69cb278be6f620c4043379c0f8b3efc6654339e4cf6d4a7a9c077931" exitCode=0 Oct 30 05:30:45.615361 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.615328 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" event={"ID":"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d","Type":"ContainerDied","Data":"0de7de9f69cb278be6f620c4043379c0f8b3efc6654339e4cf6d4a7a9c077931"} Oct 30 05:30:45.615682 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.615660 5072 scope.go:122] "RemoveContainer" containerID="0de7de9f69cb278be6f620c4043379c0f8b3efc6654339e4cf6d4a7a9c077931" Oct 30 05:30:45.617329 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.617311 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bnfpg" event={"ID":"0ee3f1bf-665e-4220-ac90-5d0b041d7077","Type":"ContainerStarted","Data":"a80c97a14c29815fed017614180bbe1178d14909fa3fcf61315b9df1b77dbe33"} Oct 30 05:30:45.617383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:45.617361 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bnfpg" event={"ID":"0ee3f1bf-665e-4220-ac90-5d0b041d7077","Type":"ContainerStarted","Data":"45c76206e6ef5252dac59188cc7b0ca3a7e18454e28ea6a19d57000f77cd1575"} Oct 30 05:30:46.145177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:46.145143 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:46.622755 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:46.622721 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jptrr" event={"ID":"4ab23bbe-53ac-43bd-a2a2-d8ca85cf6c7d","Type":"ContainerStarted","Data":"b927e01acfe929dadb5e24a33bd3222026be77c216c1e39a299a664c88dbd978"} Oct 30 05:30:46.624672 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:46.624651 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" event={"ID":"60d71bc3-a236-4305-baeb-df6403b52f43","Type":"ContainerStarted","Data":"158451f32fc13d63a05b6f11106858543b32a385110afd942e91209513280595"} Oct 30 05:30:46.910130 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:46.910044 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:30:47.301887 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:47.301808 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:47.302038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:47.301949 5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:30:47.303167 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:47.303144 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-54dbcc76d9-x7c5h" Oct 30 05:30:48.193264 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:48.193237 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:49.317126 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.317088 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:49.317126 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.317130 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:49.317706 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.317158 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:49.317706 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.317215 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:49.319727 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.319704 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95b26dcb-4282-498a-b278-4c7bb6ac11a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlm2v\" (UID: \"95b26dcb-4282-498a-b278-4c7bb6ac11a1\") " pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:49.319854 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.319775 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"image-registry-cc48cb9d8-jzp2t\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:49.320181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.320166 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"image-registry-869865c857-td99r\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:49.320238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.320179 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e52c123-4401-4329-8058-5479f3680281-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gx2pw\" (UID: \"1e52c123-4401-4329-8058-5479f3680281\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:49.325117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.325098 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:49.327265 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.327245 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" Oct 30 05:30:49.339429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.339405 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:49.343542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.343523 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tlm2v" Oct 30 05:30:49.644426 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.642307 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw"] Oct 30 05:30:49.654646 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.654624 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869865c857-td99r"] Oct 30 05:30:49.660645 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:49.660609 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd827f0_3153_4b94_8326_8c6fd0744f6b.slice/crio-191a33a9a7eef4281403d2e80939e756d127c3b26f344e3cdd493ef02c200adb WatchSource:0}: Error finding container 191a33a9a7eef4281403d2e80939e756d127c3b26f344e3cdd493ef02c200adb: Status 404 returned error can't find the container with id 191a33a9a7eef4281403d2e80939e756d127c3b26f344e3cdd493ef02c200adb Oct 30 05:30:49.667188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.667165 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc48cb9d8-jzp2t"] Oct 30 05:30:49.675520 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:49.675489 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35863a55_5422_4304_ad71_afddab4d4807.slice/crio-e2a94d3624b4dca6450e96d861f455fe171c477bb75c5f41fcee9ef62578e821 WatchSource:0}: Error finding container e2a94d3624b4dca6450e96d861f455fe171c477bb75c5f41fcee9ef62578e821: Status 404 returned error can't find the container with id e2a94d3624b4dca6450e96d861f455fe171c477bb75c5f41fcee9ef62578e821 Oct 30 05:30:49.679419 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.679309 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tlm2v"] Oct 30 05:30:49.680532 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:49.680505 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b26dcb_4282_498a_b278_4c7bb6ac11a1.slice/crio-a54b0319a85e0bf465d707bd2112db720a415b719d193cc0f34a1bae35a78b46 WatchSource:0}: Error finding container a54b0319a85e0bf465d707bd2112db720a415b719d193cc0f34a1bae35a78b46: Status 404 returned error can't find the container with id a54b0319a85e0bf465d707bd2112db720a415b719d193cc0f34a1bae35a78b46 Oct 30 05:30:49.747887 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.747861 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:49.748025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.747977 5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:30:49.754921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:49.754651 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c9p4j" Oct 30 05:30:50.107804 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.107779 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:30:50.107988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.107973 5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:30:50.643098 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.643019 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869865c857-td99r" event={"ID":"2fd827f0-3153-4b94-8326-8c6fd0744f6b","Type":"ContainerStarted","Data":"c200e18e14849ed2120440d3f15082549d522eadbad0a539d1ac32c5f38c5db2"} Oct 30 05:30:50.643098 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.643083 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869865c857-td99r" event={"ID":"2fd827f0-3153-4b94-8326-8c6fd0744f6b","Type":"ContainerStarted","Data":"191a33a9a7eef4281403d2e80939e756d127c3b26f344e3cdd493ef02c200adb"} Oct 30 05:30:50.644419 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.644387 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" event={"ID":"1e52c123-4401-4329-8058-5479f3680281","Type":"ContainerStarted","Data":"ca826377ebffc207a33ae26e2ca951fa9b82cb7bfe7503e174f0e0c21d0296b0"} Oct 30 05:30:50.646557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.646529 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlm2v" event={"ID":"95b26dcb-4282-498a-b278-4c7bb6ac11a1","Type":"ContainerStarted","Data":"c1f218a13537823e793bee26e1c77055cccf74ccd7f6a66fb8bdad400f795d53"} Oct 30 05:30:50.646679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.646612 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlm2v" event={"ID":"95b26dcb-4282-498a-b278-4c7bb6ac11a1","Type":"ContainerStarted","Data":"507b11099fca0744c22b7eb5c7505ac2ed6f639e0c587d5da3fe6cfb863110d8"} Oct 30 05:30:50.646679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.646644 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlm2v" event={"ID":"95b26dcb-4282-498a-b278-4c7bb6ac11a1","Type":"ContainerStarted","Data":"a54b0319a85e0bf465d707bd2112db720a415b719d193cc0f34a1bae35a78b46"} Oct 30 05:30:50.648685 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.648603 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" event={"ID":"35863a55-5422-4304-ad71-afddab4d4807","Type":"ContainerStarted","Data":"e935fa53dff8d7256649f1ed9282e0d9a64d97368839f2b1803c961777b3ff56"} Oct 30 05:30:50.648685 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:50.648654 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" event={"ID":"35863a55-5422-4304-ad71-afddab4d4807","Type":"ContainerStarted","Data":"e2a94d3624b4dca6450e96d861f455fe171c477bb75c5f41fcee9ef62578e821"} Oct 30 05:30:51.655386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:51.655351 5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:30:51.656573 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:51.656542 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" event={"ID":"1e52c123-4401-4329-8058-5479f3680281","Type":"ContainerStarted","Data":"48dff93fbca8271d7f2fd8b8cf228883d1bbdbae9237b773d810d0c96baa5f05"} Oct 30 05:30:51.656699 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:51.656633 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gx2pw" event={"ID":"1e52c123-4401-4329-8058-5479f3680281","Type":"ContainerStarted","Data":"79f07f01d1604318135fae297cc346b7cf5d4ddd2cde900f4560f8bab2dd5225"} Oct 30 05:30:51.656765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:51.656703 5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:30:52.399567 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:52.399536 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:30:52.661212 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:52.661117 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlm2v" event={"ID":"95b26dcb-4282-498a-b278-4c7bb6ac11a1","Type":"ContainerStarted","Data":"102c8d36613643a05cbc3cfdcc05fb6a7dc39b442a9b2f4b8c84e643fb4fdc18"} Oct 30 05:30:55.380846 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:55.380815 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:30:56.631038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:56.631014 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bnfpg" Oct 30 05:30:58.657874 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.657845 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-nmfjn"] Oct 30 05:30:58.663227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.663206 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:30:58.670765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.670685 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79b5fc4d8f-m6vsb"] Oct 30 05:30:58.674739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.674707 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.675498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.675411 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cc48cb9d8-jzp2t"] Oct 30 05:30:58.685737 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.685717 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 05:30:58.691611 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.691589 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79b5fc4d8f-m6vsb"] Oct 30 05:30:58.692407 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.692384 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nmfjn"] Oct 30 05:30:58.699122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.699086 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-serving-cert\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.699122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.699114 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-console-config\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.699266 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.699188 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-oauth-serving-cert\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.699266 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.699240 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50e77776-459b-434f-bc0a-49f3afce15e9-tmp\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.699336 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.699275 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k258\" (UniqueName: \"kubernetes.io/projected/87abc062-aaa6-45bf-b031-647053913cea-kube-api-access-9k258\") pod \"downloads-6dd789c8b7-nmfjn\" (UID: \"87abc062-aaa6-45bf-b031-647053913cea\") " pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:30:58.699336 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.699329 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2x29\" (UniqueName: \"kubernetes.io/projected/50e77776-459b-434f-bc0a-49f3afce15e9-kube-api-access-h2x29\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.699421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.699370 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-service-ca\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.699421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.699387 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-oauth-config\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.699421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.699411 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87abc062-aaa6-45bf-b031-647053913cea-tmp\") pod \"downloads-6dd789c8b7-nmfjn\" (UID: \"87abc062-aaa6-45bf-b031-647053913cea\") " pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:30:58.702191 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.702174 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 05:30:58.702584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.702566 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-pgths\"" Oct 30 05:30:58.709928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.709909 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 05:30:58.710017 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.709934 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 05:30:58.710017 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.709976 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 05:30:58.738686 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.738663 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz"] Oct 30 05:30:58.742072 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.742046 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" Oct 30 05:30:58.763437 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.763413 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz"] Oct 30 05:30:58.792420 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.792389 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 05:30:58.792420 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.792411 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 05:30:58.792627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.792389 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6tlc5\"" Oct 30 05:30:58.792627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.792505 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 05:30:58.792627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.792547 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hb5vm\"" Oct 30 05:30:58.799877 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.799856 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-oauth-serving-cert\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.799993 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.799888 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50e77776-459b-434f-bc0a-49f3afce15e9-tmp\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.799993 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.799927 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k258\" (UniqueName: \"kubernetes.io/projected/87abc062-aaa6-45bf-b031-647053913cea-kube-api-access-9k258\") pod \"downloads-6dd789c8b7-nmfjn\" (UID: \"87abc062-aaa6-45bf-b031-647053913cea\") " pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:30:58.799993 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.799956 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2x29\" (UniqueName: \"kubernetes.io/projected/50e77776-459b-434f-bc0a-49f3afce15e9-kube-api-access-h2x29\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.800144 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.799995 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3f1d1c5d-92b2-4620-a917-47535eb92aa2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xl9cz\" (UID: \"3f1d1c5d-92b2-4620-a917-47535eb92aa2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" Oct 30 05:30:58.800204 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800161 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-service-ca\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.800257 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800203 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-oauth-config\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.800315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800251 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87abc062-aaa6-45bf-b031-647053913cea-tmp\") pod \"downloads-6dd789c8b7-nmfjn\" (UID: \"87abc062-aaa6-45bf-b031-647053913cea\") " pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:30:58.800315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800300 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-serving-cert\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.800408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800325 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-console-config\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.800408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800329 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50e77776-459b-434f-bc0a-49f3afce15e9-tmp\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.800707 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800685 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-oauth-serving-cert\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.800822 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800712 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87abc062-aaa6-45bf-b031-647053913cea-tmp\") pod \"downloads-6dd789c8b7-nmfjn\" (UID: \"87abc062-aaa6-45bf-b031-647053913cea\") " pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:30:58.800895 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800862 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-service-ca\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.800992 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.800973 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-console-config\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.802898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.802876 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-oauth-config\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.802994 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.802897 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-serving-cert\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.827842 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.827812 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k258\" (UniqueName: \"kubernetes.io/projected/87abc062-aaa6-45bf-b031-647053913cea-kube-api-access-9k258\") pod \"downloads-6dd789c8b7-nmfjn\" (UID: \"87abc062-aaa6-45bf-b031-647053913cea\") " pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:30:58.827982 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.827925 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2x29\" (UniqueName: \"kubernetes.io/projected/50e77776-459b-434f-bc0a-49f3afce15e9-kube-api-access-h2x29\") pod \"console-79b5fc4d8f-m6vsb\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:58.835729 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.835706 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-845d4d8d7d-6df2c"] Oct 30 05:30:58.841211 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.841193 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:58.842698 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.842676 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-675c5fcc5b-qj47v"] Oct 30 05:30:58.846138 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.846120 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:58.870757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.870728 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 05:30:58.873557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.873530 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845d4d8d7d-6df2c"] Oct 30 05:30:58.873720 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.873706 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675c5fcc5b-qj47v"] Oct 30 05:30:58.901303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901277 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlwts\" (UniqueName: \"kubernetes.io/projected/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-kube-api-access-wlwts\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:58.901417 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901311 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mrh\" (UniqueName: \"kubernetes.io/projected/34204eac-55de-46b5-84d0-d6c1c822ce87-kube-api-access-g9mrh\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:58.901417 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901404 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3f1d1c5d-92b2-4620-a917-47535eb92aa2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xl9cz\" (UID: \"3f1d1c5d-92b2-4620-a917-47535eb92aa2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" Oct 30 05:30:58.901542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901433 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-config\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:58.901542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901449 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34204eac-55de-46b5-84d0-d6c1c822ce87-trusted-ca\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:58.901542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901498 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-oauth-config\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:58.901642 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901547 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-tmp\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:58.901642 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901584 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-trusted-ca-bundle\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:58.901642 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901610 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34204eac-55de-46b5-84d0-d6c1c822ce87-installation-pull-secrets\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:58.901745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901679 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34204eac-55de-46b5-84d0-d6c1c822ce87-registry-tls\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:58.901745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901723 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-serving-cert\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:58.901813 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901789 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34204eac-55de-46b5-84d0-d6c1c822ce87-image-registry-private-configuration\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:58.901847 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901824 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-oauth-serving-cert\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:58.901885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901854 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-service-ca\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:58.901922 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901884 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34204eac-55de-46b5-84d0-d6c1c822ce87-ca-trust-extracted\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:58.901922 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901904 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34204eac-55de-46b5-84d0-d6c1c822ce87-registry-certificates\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:58.902008 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.901930 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34204eac-55de-46b5-84d0-d6c1c822ce87-bound-sa-token\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:58.904048 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.904028 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3f1d1c5d-92b2-4620-a917-47535eb92aa2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xl9cz\" (UID: \"3f1d1c5d-92b2-4620-a917-47535eb92aa2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" Oct 30 05:30:58.978289 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.978252 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:30:58.997417 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:58.997387 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:30:59.002840 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.002812 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-serving-cert\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.002972 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.002857 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34204eac-55de-46b5-84d0-d6c1c822ce87-image-registry-private-configuration\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.002972 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.002886 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-oauth-serving-cert\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.002972 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.002913 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-service-ca\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.002972 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.002935 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34204eac-55de-46b5-84d0-d6c1c822ce87-ca-trust-extracted\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.002972 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.002964 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34204eac-55de-46b5-84d0-d6c1c822ce87-registry-certificates\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.002986 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34204eac-55de-46b5-84d0-d6c1c822ce87-bound-sa-token\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003013 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwts\" (UniqueName: \"kubernetes.io/projected/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-kube-api-access-wlwts\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003033 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mrh\" (UniqueName: \"kubernetes.io/projected/34204eac-55de-46b5-84d0-d6c1c822ce87-kube-api-access-g9mrh\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003077 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-config\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003099 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34204eac-55de-46b5-84d0-d6c1c822ce87-trusted-ca\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003149 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-oauth-config\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003171 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-tmp\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003202 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-trusted-ca-bundle\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003235 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34204eac-55de-46b5-84d0-d6c1c822ce87-installation-pull-secrets\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.003274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003261 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34204eac-55de-46b5-84d0-d6c1c822ce87-registry-tls\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.003965 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.003941 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34204eac-55de-46b5-84d0-d6c1c822ce87-registry-certificates\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.004809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.004780 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34204eac-55de-46b5-84d0-d6c1c822ce87-trusted-ca\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.005083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.005062 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-service-ca\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.005777 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.005754 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-tmp\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.005777 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.005770 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-config\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.006083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.006066 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-serving-cert\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.006278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.006255 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34204eac-55de-46b5-84d0-d6c1c822ce87-registry-tls\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.006538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.006497 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-trusted-ca-bundle\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.006807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.006625 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-oauth-serving-cert\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.006807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.006747 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34204eac-55de-46b5-84d0-d6c1c822ce87-ca-trust-extracted\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.008897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.008874 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34204eac-55de-46b5-84d0-d6c1c822ce87-installation-pull-secrets\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.009354 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.009336 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-oauth-config\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.009448 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.009427 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34204eac-55de-46b5-84d0-d6c1c822ce87-image-registry-private-configuration\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.029801 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.029739 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwts\" (UniqueName: \"kubernetes.io/projected/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-kube-api-access-wlwts\") pod \"console-845d4d8d7d-6df2c\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.031680 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.031654 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34204eac-55de-46b5-84d0-d6c1c822ce87-bound-sa-token\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.031821 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.031710 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mrh\" (UniqueName: \"kubernetes.io/projected/34204eac-55de-46b5-84d0-d6c1c822ce87-kube-api-access-g9mrh\") pod \"image-registry-675c5fcc5b-qj47v\" (UID: \"34204eac-55de-46b5-84d0-d6c1c822ce87\") " pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.055015 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.054983 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" Oct 30 05:30:59.155043 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.155007 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:30:59.160186 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.160170 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.295365 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.295334 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nmfjn"] Oct 30 05:30:59.296581 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:59.296559 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87abc062_aaa6_45bf_b031_647053913cea.slice/crio-667eaa24ca4eaa0c98a224d94bbe992bdfcff87f512092d6a3b4dd8efe05a2e6 WatchSource:0}: Error finding container 667eaa24ca4eaa0c98a224d94bbe992bdfcff87f512092d6a3b4dd8efe05a2e6: Status 404 returned error can't find the container with id 667eaa24ca4eaa0c98a224d94bbe992bdfcff87f512092d6a3b4dd8efe05a2e6 Oct 30 05:30:59.315284 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.315179 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79b5fc4d8f-m6vsb"] Oct 30 05:30:59.316125 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:59.316101 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e77776_459b_434f_bc0a_49f3afce15e9.slice/crio-713b0618b2a9e64c9b557cf6d9975699ed34311937d38a0e7a0a902beed9de2c WatchSource:0}: Error finding container 713b0618b2a9e64c9b557cf6d9975699ed34311937d38a0e7a0a902beed9de2c: Status 404 returned error can't find the container with id 713b0618b2a9e64c9b557cf6d9975699ed34311937d38a0e7a0a902beed9de2c Oct 30 05:30:59.358323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.358297 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz"] Oct 30 05:30:59.359907 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:59.359881 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1d1c5d_92b2_4620_a917_47535eb92aa2.slice/crio-e038e4e727acf9a845df9ad2aff8df20f918c4e5b08a325f01d67f4aa23b02c7 WatchSource:0}: Error finding container e038e4e727acf9a845df9ad2aff8df20f918c4e5b08a325f01d67f4aa23b02c7: Status 404 returned error can't find the container with id e038e4e727acf9a845df9ad2aff8df20f918c4e5b08a325f01d67f4aa23b02c7 Oct 30 05:30:59.461047 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.461021 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845d4d8d7d-6df2c"] Oct 30 05:30:59.461940 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:59.461914 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f4df7d_3912_4d92_a0fb_83e936b8ff9f.slice/crio-702ffc54c45e8f19b65dbdc7225ea81c998843da46f36aa801a3c2c7d9104bac WatchSource:0}: Error finding container 702ffc54c45e8f19b65dbdc7225ea81c998843da46f36aa801a3c2c7d9104bac: Status 404 returned error can't find the container with id 702ffc54c45e8f19b65dbdc7225ea81c998843da46f36aa801a3c2c7d9104bac Oct 30 05:30:59.467829 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.467807 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675c5fcc5b-qj47v"] Oct 30 05:30:59.469885 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:30:59.469863 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34204eac_55de_46b5_84d0_d6c1c822ce87.slice/crio-395c8b9de9180431d08dc12c0d3cc6b43af1ca2a727e53dd8ec2cc94bc218549 WatchSource:0}: Error finding container 395c8b9de9180431d08dc12c0d3cc6b43af1ca2a727e53dd8ec2cc94bc218549: Status 404 returned error can't find the container with id 395c8b9de9180431d08dc12c0d3cc6b43af1ca2a727e53dd8ec2cc94bc218549 Oct 30 05:30:59.701316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.701235 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b5fc4d8f-m6vsb" event={"ID":"50e77776-459b-434f-bc0a-49f3afce15e9","Type":"ContainerStarted","Data":"713b0618b2a9e64c9b557cf6d9975699ed34311937d38a0e7a0a902beed9de2c"} Oct 30 05:30:59.702762 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.702735 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" event={"ID":"34204eac-55de-46b5-84d0-d6c1c822ce87","Type":"ContainerStarted","Data":"7cd99112104d869fcd6ca913a2c50d54cfebf9f73aa40d48c9bcacfa5c6a48eb"} Oct 30 05:30:59.703041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.702778 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" event={"ID":"34204eac-55de-46b5-84d0-d6c1c822ce87","Type":"ContainerStarted","Data":"395c8b9de9180431d08dc12c0d3cc6b43af1ca2a727e53dd8ec2cc94bc218549"} Oct 30 05:30:59.703041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.702853 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:30:59.704310 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.704277 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845d4d8d7d-6df2c" event={"ID":"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f","Type":"ContainerStarted","Data":"702ffc54c45e8f19b65dbdc7225ea81c998843da46f36aa801a3c2c7d9104bac"} Oct 30 05:30:59.705708 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.705689 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" event={"ID":"3f1d1c5d-92b2-4620-a917-47535eb92aa2","Type":"ContainerStarted","Data":"e038e4e727acf9a845df9ad2aff8df20f918c4e5b08a325f01d67f4aa23b02c7"} Oct 30 05:30:59.707041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.707022 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nmfjn" event={"ID":"87abc062-aaa6-45bf-b031-647053913cea","Type":"ContainerStarted","Data":"667eaa24ca4eaa0c98a224d94bbe992bdfcff87f512092d6a3b4dd8efe05a2e6"} Oct 30 05:30:59.748708 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:30:59.748647 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" podStartSLOduration=1.748630312 podStartE2EDuration="1.748630312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:59.747822474 +0000 UTC m=+19.351511767" watchObservedRunningTime="2026-10-30 05:30:59.748630312 +0000 UTC m=+19.352319572" Oct 30 05:31:00.717968 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:00.717918 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" event={"ID":"3f1d1c5d-92b2-4620-a917-47535eb92aa2","Type":"ContainerStarted","Data":"35d8ebd68728fffbe90910037be698f3af21692999ce423235f278b7fbc9206d"} Oct 30 05:31:00.718446 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:00.718164 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" Oct 30 05:31:00.726014 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:00.725991 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" Oct 30 05:31:00.734935 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:31:00.734905 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:31:00.770510 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:00.770443 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xl9cz" podStartSLOduration=1.58449813 podStartE2EDuration="2.770429387s" totalImagesPullingTime="1.185931257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:00.758242507 +0000 UTC m=+20.361931765" watchObservedRunningTime="2026-10-30 05:31:00.770429387 +0000 UTC m=+20.374118638" Oct 30 05:31:03.902152 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:03.902120 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d2qsz" Oct 30 05:31:04.738153 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:04.738115 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b5fc4d8f-m6vsb" event={"ID":"50e77776-459b-434f-bc0a-49f3afce15e9","Type":"ContainerStarted","Data":"9ad4c37354540559fe304892a5d5003799028af08908a774fe8c543389a7e117"} Oct 30 05:31:04.740156 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:04.740117 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845d4d8d7d-6df2c" event={"ID":"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f","Type":"ContainerStarted","Data":"9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4"} Oct 30 05:31:04.781653 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:04.781598 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79b5fc4d8f-m6vsb" podStartSLOduration=2.413567293 podStartE2EDuration="6.781586749s" totalImagesPullingTime="4.368019456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:04.780816321 +0000 UTC m=+24.384505580" watchObservedRunningTime="2026-10-30 05:31:04.781586749 +0000 UTC m=+24.385276006" Oct 30 05:31:04.824448 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:04.824395 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-845d4d8d7d-6df2c" podStartSLOduration=2.588325375 podStartE2EDuration="6.824381289s" totalImagesPullingTime="4.236055914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:04.823308588 +0000 UTC m=+24.426997845" watchObservedRunningTime="2026-10-30 05:31:04.824381289 +0000 UTC m=+24.428070545" Oct 30 05:31:05.965062 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:05.965026 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jmw7r"] Oct 30 05:31:06.003883 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.003838 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.043958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.043928 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:31:06.043958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.043935 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:31:06.080702 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.080668 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-sys\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.080702 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.080706 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-textfile\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.080941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.080732 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-wtmp\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.080941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.080803 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-metrics-client-ca\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.081041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.080936 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5cqw\" (UniqueName: \"kubernetes.io/projected/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-kube-api-access-r5cqw\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.081041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.080984 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.081152 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.081051 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.081152 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.081118 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-root\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.081152 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.081142 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-tls\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.096532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.096504 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:31:06.096752 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.096733 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:31:06.096857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.096740 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2znb\"" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.182629 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.182699 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.182755 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-root\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.182786 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-tls\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.182822 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-sys\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.182844 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-textfile\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.182876 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-wtmp\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.182898 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-metrics-client-ca\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.182929 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5cqw\" (UniqueName: \"kubernetes.io/projected/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-kube-api-access-r5cqw\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.183115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.183089 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-sys\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.184451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.183148 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-root\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.184451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.183272 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-wtmp\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.184451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.183960 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-metrics-client-ca\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.184451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.184437 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-textfile\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.185021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.184978 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.187402 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.186066 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.187879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.187859 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-node-exporter-tls\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.210388 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.210350 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5cqw\" (UniqueName: \"kubernetes.io/projected/d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5-kube-api-access-r5cqw\") pod \"node-exporter-jmw7r\" (UID: \"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5\") " pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.377374 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.377336 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jmw7r" Oct 30 05:31:06.416057 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:31:06.416023 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89fa6e9_a11a_4b9f_ba96_5c7f8b0e26f5.slice/crio-89f8df2eef7d37b5c311264dcfa9da045329b3a27dcefa0d8d5af65c5813c3d0 WatchSource:0}: Error finding container 89f8df2eef7d37b5c311264dcfa9da045329b3a27dcefa0d8d5af65c5813c3d0: Status 404 returned error can't find the container with id 89f8df2eef7d37b5c311264dcfa9da045329b3a27dcefa0d8d5af65c5813c3d0 Oct 30 05:31:06.752123 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.752042 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jmw7r" event={"ID":"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5","Type":"ContainerStarted","Data":"89f8df2eef7d37b5c311264dcfa9da045329b3a27dcefa0d8d5af65c5813c3d0"} Oct 30 05:31:06.914004 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:06.913975 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bmv92" Oct 30 05:31:08.682535 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:08.682505 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:31:08.763996 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:08.763956 5072 generic.go:356] "Generic (PLEG): container finished" podID="d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5" containerID="13e20f59a923015a456a11b4667cbdd089430283026a643f49ea1963aae0d9e2" exitCode=0 Oct 30 05:31:08.764171 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:08.764116 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jmw7r" event={"ID":"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5","Type":"ContainerDied","Data":"13e20f59a923015a456a11b4667cbdd089430283026a643f49ea1963aae0d9e2"} Oct 30 05:31:08.997596 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:08.997569 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:31:08.997773 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:08.997640 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:31:09.003252 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.003230 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:31:09.156025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.155993 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:31:09.156179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.156055 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:31:09.162189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.162162 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:31:09.771971 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.771933 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jmw7r" event={"ID":"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5","Type":"ContainerStarted","Data":"2c5b1776f1bf496683b3bcc25614a1bd95bc289ee6d0572bb0b6b60ab9e063c7"} Oct 30 05:31:09.772415 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.772006 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jmw7r" event={"ID":"d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5","Type":"ContainerStarted","Data":"0b1b3965eaa817ce7d51799cefeb94b1a03568bbd9d6ac2e49e58ef60abceebb"} Oct 30 05:31:09.777385 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.777360 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:31:09.777980 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.777929 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:31:09.832139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.832083 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jmw7r" podStartSLOduration=3.250675304 podStartE2EDuration="4.832065519s" totalImagesPullingTime="1.488496759s" totalInitContainerRuntime="92.893456ms" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:09.829777069 +0000 UTC m=+29.433466326" watchObservedRunningTime="2026-10-30 05:31:09.832065519 +0000 UTC m=+29.435754777" Oct 30 05:31:09.961861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:09.961833 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79b5fc4d8f-m6vsb"] Oct 30 05:31:12.367110 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.367078 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5445c766fd-qj666"] Oct 30 05:31:12.402293 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.402247 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5445c766fd-qj666"] Oct 30 05:31:12.402495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.402384 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.444987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.444945 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-serving-cert\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.444987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.444983 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d12839-c1e9-481d-9b3e-65065606dd3d-tmp\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.445228 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.445011 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-oauth-serving-cert\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.445228 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.445122 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-trusted-ca-bundle\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.445228 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.445185 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vf7\" (UniqueName: \"kubernetes.io/projected/91d12839-c1e9-481d-9b3e-65065606dd3d-kube-api-access-k7vf7\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.445388 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.445288 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-service-ca\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.445388 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.445352 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-oauth-config\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.445388 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.445372 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-console-config\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.546571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.546535 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-console-config\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.546779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.546608 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-serving-cert\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.546779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.546634 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d12839-c1e9-481d-9b3e-65065606dd3d-tmp\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.546779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.546664 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-oauth-serving-cert\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.546779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.546690 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-trusted-ca-bundle\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.547013 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.546837 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vf7\" (UniqueName: \"kubernetes.io/projected/91d12839-c1e9-481d-9b3e-65065606dd3d-kube-api-access-k7vf7\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.547013 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.546893 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-service-ca\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.547013 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.546979 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-oauth-config\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.547158 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.547102 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d12839-c1e9-481d-9b3e-65065606dd3d-tmp\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.547405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.547382 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-console-config\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.547514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.547417 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-oauth-serving-cert\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.547838 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.547814 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-trusted-ca-bundle\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.547945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.547841 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-service-ca\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.549854 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.549832 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-oauth-config\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.549968 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.549952 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-serving-cert\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.571030 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.571000 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vf7\" (UniqueName: \"kubernetes.io/projected/91d12839-c1e9-481d-9b3e-65065606dd3d-kube-api-access-k7vf7\") pod \"console-5445c766fd-qj666\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:12.666352 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.666269 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:31:12.732737 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:12.732694 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:13.070549 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:13.070521 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5445c766fd-qj666"] Oct 30 05:31:13.072728 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:31:13.072702 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d12839_c1e9_481d_9b3e_65065606dd3d.slice/crio-94ece3bd92bab72db28e70eb144f2624cf40e932db21ba5e3abea13ed1caed47 WatchSource:0}: Error finding container 94ece3bd92bab72db28e70eb144f2624cf40e932db21ba5e3abea13ed1caed47: Status 404 returned error can't find the container with id 94ece3bd92bab72db28e70eb144f2624cf40e932db21ba5e3abea13ed1caed47 Oct 30 05:31:13.797111 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:13.797075 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5445c766fd-qj666" event={"ID":"91d12839-c1e9-481d-9b3e-65065606dd3d","Type":"ContainerStarted","Data":"1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7"} Oct 30 05:31:13.797572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:13.797126 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5445c766fd-qj666" event={"ID":"91d12839-c1e9-481d-9b3e-65065606dd3d","Type":"ContainerStarted","Data":"94ece3bd92bab72db28e70eb144f2624cf40e932db21ba5e3abea13ed1caed47"} Oct 30 05:31:13.840386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:13.840329 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5445c766fd-qj666" podStartSLOduration=1.84030261 podStartE2EDuration="1.84030261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:13.839786598 +0000 UTC m=+33.443475857" watchObservedRunningTime="2026-10-30 05:31:13.84030261 +0000 UTC m=+33.443991871" Oct 30 05:31:16.815005 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:16.814963 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nmfjn" event={"ID":"87abc062-aaa6-45bf-b031-647053913cea","Type":"ContainerStarted","Data":"27924fcc44e80674c2da7ae787e30dca06b2e303dd7f54dd9ac51bada4be4246"} Oct 30 05:31:16.815586 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:16.815196 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:31:16.827662 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:16.827635 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-nmfjn" Oct 30 05:31:16.882691 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:16.882637 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-nmfjn" podStartSLOduration=1.956406823 podStartE2EDuration="18.882621867s" totalImagesPullingTime="16.926215044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:16.869362391 +0000 UTC m=+36.473051641" watchObservedRunningTime="2026-10-30 05:31:16.882621867 +0000 UTC m=+36.486311169" Oct 30 05:31:20.722049 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:20.721957 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-675c5fcc5b-qj47v" Oct 30 05:31:20.818452 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:20.818417 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-869865c857-td99r"] Oct 30 05:31:22.733447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:22.733408 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:22.734071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:22.733502 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:22.739142 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:22.739119 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:22.864179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:22.864145 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:31:22.944724 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:22.944686 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845d4d8d7d-6df2c"] Oct 30 05:31:23.700510 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:23.700411 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" podUID="35863a55-5422-4304-ad71-afddab4d4807" containerName="registry" containerID="cri-o://e935fa53dff8d7256649f1ed9282e0d9a64d97368839f2b1803c961777b3ff56" gracePeriod=30 Oct 30 05:31:23.867907 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:23.867866 5072 generic.go:356] "Generic (PLEG): container finished" podID="35863a55-5422-4304-ad71-afddab4d4807" containerID="e935fa53dff8d7256649f1ed9282e0d9a64d97368839f2b1803c961777b3ff56" exitCode=0 Oct 30 05:31:23.868622 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:23.868598 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" event={"ID":"35863a55-5422-4304-ad71-afddab4d4807","Type":"ContainerDied","Data":"e935fa53dff8d7256649f1ed9282e0d9a64d97368839f2b1803c961777b3ff56"} Oct 30 05:31:24.012727 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.012696 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:31:24.166271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.166239 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration\") pod \"35863a55-5422-4304-ad71-afddab4d4807\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " Oct 30 05:31:24.166431 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.166293 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m\") pod \"35863a55-5422-4304-ad71-afddab4d4807\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " Oct 30 05:31:24.166431 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.166338 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca\") pod \"35863a55-5422-4304-ad71-afddab4d4807\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " Oct 30 05:31:24.166431 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.166390 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token\") pod \"35863a55-5422-4304-ad71-afddab4d4807\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " Oct 30 05:31:24.166431 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.166422 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates\") pod \"35863a55-5422-4304-ad71-afddab4d4807\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " Oct 30 05:31:24.166713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.166478 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted\") pod \"35863a55-5422-4304-ad71-afddab4d4807\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " Oct 30 05:31:24.166713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.166521 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets\") pod \"35863a55-5422-4304-ad71-afddab4d4807\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " Oct 30 05:31:24.166713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.166556 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") pod \"35863a55-5422-4304-ad71-afddab4d4807\" (UID: \"35863a55-5422-4304-ad71-afddab4d4807\") " Oct 30 05:31:24.171400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.167155 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca" pod "35863a55-5422-4304-ad71-afddab4d4807" (UID: "35863a55-5422-4304-ad71-afddab4d4807"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:24.171400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.169836 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates" pod "35863a55-5422-4304-ad71-afddab4d4807" (UID: "35863a55-5422-4304-ad71-afddab4d4807"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:24.171743 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.171699 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets" pod "35863a55-5422-4304-ad71-afddab4d4807" (UID: "35863a55-5422-4304-ad71-afddab4d4807"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:24.171875 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.171813 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration" pod "35863a55-5422-4304-ad71-afddab4d4807" (UID: "35863a55-5422-4304-ad71-afddab4d4807"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:24.175707 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.175679 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m" pod "35863a55-5422-4304-ad71-afddab4d4807" (UID: "35863a55-5422-4304-ad71-afddab4d4807"). InnerVolumeSpecName "kube-api-access-gxd2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:24.175927 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.175892 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token" pod "35863a55-5422-4304-ad71-afddab4d4807" (UID: "35863a55-5422-4304-ad71-afddab4d4807"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:24.176006 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.175942 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls" pod "35863a55-5422-4304-ad71-afddab4d4807" (UID: "35863a55-5422-4304-ad71-afddab4d4807"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:24.176480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.176438 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted" pod "35863a55-5422-4304-ad71-afddab4d4807" (UID: "35863a55-5422-4304-ad71-afddab4d4807"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:24.267939 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.267862 5072 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-registry-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:24.267939 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.267900 5072 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-image-registry-private-configuration\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:24.267939 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.267917 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxd2m\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-kube-api-access-gxd2m\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:24.267939 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.267932 5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-trusted-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:24.268208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.267945 5072 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35863a55-5422-4304-ad71-afddab4d4807-bound-sa-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:24.268208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.267956 5072 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35863a55-5422-4304-ad71-afddab4d4807-registry-certificates\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:24.268208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.267964 5072 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35863a55-5422-4304-ad71-afddab4d4807-ca-trust-extracted\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:24.268208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.267973 5072 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35863a55-5422-4304-ad71-afddab4d4807-installation-pull-secrets\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:24.875188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.875154 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" Oct 30 05:31:24.875188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.875163 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc48cb9d8-jzp2t" event={"ID":"35863a55-5422-4304-ad71-afddab4d4807","Type":"ContainerDied","Data":"e2a94d3624b4dca6450e96d861f455fe171c477bb75c5f41fcee9ef62578e821"} Oct 30 05:31:24.875802 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.875229 5072 scope.go:122] "RemoveContainer" containerID="e935fa53dff8d7256649f1ed9282e0d9a64d97368839f2b1803c961777b3ff56" Oct 30 05:31:24.940022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.939989 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cc48cb9d8-jzp2t"] Oct 30 05:31:24.952188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:24.952164 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cc48cb9d8-jzp2t"] Oct 30 05:31:26.516281 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:26.516236 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35863a55-5422-4304-ad71-afddab4d4807" path="/var/lib/kubelet/pods/35863a55-5422-4304-ad71-afddab4d4807/volumes" Oct 30 05:31:30.708843 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:31:30.708810 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:31:36.805141 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:36.805084 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-79b5fc4d8f-m6vsb" podUID="50e77776-459b-434f-bc0a-49f3afce15e9" containerName="console" containerID="cri-o://9ad4c37354540559fe304892a5d5003799028af08908a774fe8c543389a7e117" gracePeriod=15 Oct 30 05:31:36.929640 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:36.929608 5072 generic.go:356] "Generic (PLEG): container finished" podID="50e77776-459b-434f-bc0a-49f3afce15e9" containerID="9ad4c37354540559fe304892a5d5003799028af08908a774fe8c543389a7e117" exitCode=0 Oct 30 05:31:36.929794 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:36.929654 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b5fc4d8f-m6vsb" event={"ID":"50e77776-459b-434f-bc0a-49f3afce15e9","Type":"ContainerDied","Data":"9ad4c37354540559fe304892a5d5003799028af08908a774fe8c543389a7e117"} Oct 30 05:31:37.085514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.085485 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:31:37.184392 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.184356 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-oauth-config\") pod \"50e77776-459b-434f-bc0a-49f3afce15e9\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " Oct 30 05:31:37.184555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.184419 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-console-config\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-console-config\") pod \"50e77776-459b-434f-bc0a-49f3afce15e9\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " Oct 30 05:31:37.184555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.184507 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-serving-cert\") pod \"50e77776-459b-434f-bc0a-49f3afce15e9\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " Oct 30 05:31:37.184555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.184540 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e77776-459b-434f-bc0a-49f3afce15e9-kube-api-access-h2x29\" (UniqueName: \"kubernetes.io/projected/50e77776-459b-434f-bc0a-49f3afce15e9-kube-api-access-h2x29\") pod \"50e77776-459b-434f-bc0a-49f3afce15e9\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " Oct 30 05:31:37.184680 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.184578 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-oauth-serving-cert\") pod \"50e77776-459b-434f-bc0a-49f3afce15e9\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " Oct 30 05:31:37.184680 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.184632 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e77776-459b-434f-bc0a-49f3afce15e9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/50e77776-459b-434f-bc0a-49f3afce15e9-tmp\") pod \"50e77776-459b-434f-bc0a-49f3afce15e9\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " Oct 30 05:31:37.184680 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.184670 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-service-ca\") pod \"50e77776-459b-434f-bc0a-49f3afce15e9\" (UID: \"50e77776-459b-434f-bc0a-49f3afce15e9\") " Oct 30 05:31:37.184840 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.184792 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-console-config" pod "50e77776-459b-434f-bc0a-49f3afce15e9" (UID: "50e77776-459b-434f-bc0a-49f3afce15e9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:37.184983 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.184951 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e77776-459b-434f-bc0a-49f3afce15e9-tmp" pod "50e77776-459b-434f-bc0a-49f3afce15e9" (UID: "50e77776-459b-434f-bc0a-49f3afce15e9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:37.185052 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.185017 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-oauth-serving-cert" pod "50e77776-459b-434f-bc0a-49f3afce15e9" (UID: "50e77776-459b-434f-bc0a-49f3afce15e9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:37.185109 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.185060 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-service-ca" pod "50e77776-459b-434f-bc0a-49f3afce15e9" (UID: "50e77776-459b-434f-bc0a-49f3afce15e9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:37.185165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.185134 5072 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-console-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:37.185165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.185154 5072 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-oauth-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:37.185241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.185170 5072 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50e77776-459b-434f-bc0a-49f3afce15e9-tmp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:37.186846 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.186814 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-oauth-config" pod "50e77776-459b-434f-bc0a-49f3afce15e9" (UID: "50e77776-459b-434f-bc0a-49f3afce15e9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:37.186990 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.186856 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-serving-cert" pod "50e77776-459b-434f-bc0a-49f3afce15e9" (UID: "50e77776-459b-434f-bc0a-49f3afce15e9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:37.186990 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.186887 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e77776-459b-434f-bc0a-49f3afce15e9-kube-api-access-h2x29" pod "50e77776-459b-434f-bc0a-49f3afce15e9" (UID: "50e77776-459b-434f-bc0a-49f3afce15e9"). InnerVolumeSpecName "kube-api-access-h2x29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:37.286134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.286099 5072 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:37.286134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.286132 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2x29\" (UniqueName: \"kubernetes.io/projected/50e77776-459b-434f-bc0a-49f3afce15e9-kube-api-access-h2x29\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:37.286340 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.286147 5072 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50e77776-459b-434f-bc0a-49f3afce15e9-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:37.286340 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.286160 5072 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50e77776-459b-434f-bc0a-49f3afce15e9-console-oauth-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:37.935105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.935069 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b5fc4d8f-m6vsb" event={"ID":"50e77776-459b-434f-bc0a-49f3afce15e9","Type":"ContainerDied","Data":"713b0618b2a9e64c9b557cf6d9975699ed34311937d38a0e7a0a902beed9de2c"} Oct 30 05:31:37.935105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.935107 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b5fc4d8f-m6vsb" Oct 30 05:31:37.935720 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.935134 5072 scope.go:122] "RemoveContainer" containerID="9ad4c37354540559fe304892a5d5003799028af08908a774fe8c543389a7e117" Oct 30 05:31:37.989663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:37.989633 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79b5fc4d8f-m6vsb"] Oct 30 05:31:38.001152 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:38.001122 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79b5fc4d8f-m6vsb"] Oct 30 05:31:38.516383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:38.516351 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e77776-459b-434f-bc0a-49f3afce15e9" path="/var/lib/kubelet/pods/50e77776-459b-434f-bc0a-49f3afce15e9/volumes" Oct 30 05:31:44.966902 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:44.966819 5072 generic.go:356] "Generic (PLEG): container finished" podID="2f968039-9ce3-48ea-a1c6-c963206a596c" containerID="17456246d697630c019eebd1ef4c80b2629553209650a1be3da7486990e05e80" exitCode=0 Oct 30 05:31:44.967372 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:44.966899 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" event={"ID":"2f968039-9ce3-48ea-a1c6-c963206a596c","Type":"ContainerDied","Data":"17456246d697630c019eebd1ef4c80b2629553209650a1be3da7486990e05e80"} Oct 30 05:31:44.967372 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:44.967283 5072 scope.go:122] "RemoveContainer" containerID="17456246d697630c019eebd1ef4c80b2629553209650a1be3da7486990e05e80" Oct 30 05:31:45.853540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:45.853482 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-869865c857-td99r" podUID="2fd827f0-3153-4b94-8326-8c6fd0744f6b" containerName="registry" containerID="cri-o://c200e18e14849ed2120440d3f15082549d522eadbad0a539d1ac32c5f38c5db2" gracePeriod=30 Oct 30 05:31:45.972826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:45.972791 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-f2stn" event={"ID":"2f968039-9ce3-48ea-a1c6-c963206a596c","Type":"ContainerStarted","Data":"94bd3f6e2571e6729018d021da2f1c006f69fd46f821c123ec9e7762a075a3d7"} Oct 30 05:31:45.974206 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:45.974182 5072 generic.go:356] "Generic (PLEG): container finished" podID="2fd827f0-3153-4b94-8326-8c6fd0744f6b" containerID="c200e18e14849ed2120440d3f15082549d522eadbad0a539d1ac32c5f38c5db2" exitCode=0 Oct 30 05:31:45.974313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:45.974245 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869865c857-td99r" event={"ID":"2fd827f0-3153-4b94-8326-8c6fd0744f6b","Type":"ContainerDied","Data":"c200e18e14849ed2120440d3f15082549d522eadbad0a539d1ac32c5f38c5db2"} Oct 30 05:31:46.162957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.162926 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:31:46.261173 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261144 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token\") pod \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " Oct 30 05:31:46.261320 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261191 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates\") pod \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " Oct 30 05:31:46.261320 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261219 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj\") pod \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " Oct 30 05:31:46.261320 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261242 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted\") pod \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " Oct 30 05:31:46.261493 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261394 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets\") pod \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " Oct 30 05:31:46.261493 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261445 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") pod \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " Oct 30 05:31:46.261602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261508 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca\") pod \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " Oct 30 05:31:46.261602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261559 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration\") pod \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\" (UID: \"2fd827f0-3153-4b94-8326-8c6fd0744f6b\") " Oct 30 05:31:46.261694 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261627 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates" pod "2fd827f0-3153-4b94-8326-8c6fd0744f6b" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:46.261837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261819 5072 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-certificates\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:46.261951 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.261922 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca" pod "2fd827f0-3153-4b94-8326-8c6fd0744f6b" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:46.264052 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.264024 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token" pod "2fd827f0-3153-4b94-8326-8c6fd0744f6b" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:46.264237 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.264201 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj" pod "2fd827f0-3153-4b94-8326-8c6fd0744f6b" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b"). InnerVolumeSpecName "kube-api-access-dkzgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:46.264237 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.264218 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls" pod "2fd827f0-3153-4b94-8326-8c6fd0744f6b" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:46.264366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.264232 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets" pod "2fd827f0-3153-4b94-8326-8c6fd0744f6b" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:46.264502 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.264482 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration" pod "2fd827f0-3153-4b94-8326-8c6fd0744f6b" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:46.268889 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.268865 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted" pod "2fd827f0-3153-4b94-8326-8c6fd0744f6b" (UID: "2fd827f0-3153-4b94-8326-8c6fd0744f6b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:46.362628 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.362606 5072 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-installation-pull-secrets\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:46.362628 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.362629 5072 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-registry-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:46.362768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.362638 5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fd827f0-3153-4b94-8326-8c6fd0744f6b-trusted-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:46.362768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.362647 5072 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fd827f0-3153-4b94-8326-8c6fd0744f6b-image-registry-private-configuration\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:46.362768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.362657 5072 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-bound-sa-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:46.362768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.362666 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkzgj\" (UniqueName: \"kubernetes.io/projected/2fd827f0-3153-4b94-8326-8c6fd0744f6b-kube-api-access-dkzgj\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:46.362768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.362674 5072 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fd827f0-3153-4b94-8326-8c6fd0744f6b-ca-trust-extracted\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:46.981451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.981415 5072 generic.go:356] "Generic (PLEG): container finished" podID="60d71bc3-a236-4305-baeb-df6403b52f43" containerID="158451f32fc13d63a05b6f11106858543b32a385110afd942e91209513280595" exitCode=0 Oct 30 05:31:46.981945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.981501 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" event={"ID":"60d71bc3-a236-4305-baeb-df6403b52f43","Type":"ContainerDied","Data":"158451f32fc13d63a05b6f11106858543b32a385110afd942e91209513280595"} Oct 30 05:31:46.981945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.981565 5072 scope.go:122] "RemoveContainer" containerID="f6bcda01cf71459069982c449fa9598488ff34e98074ca971842a59dba058767" Oct 30 05:31:46.981945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.981916 5072 scope.go:122] "RemoveContainer" containerID="158451f32fc13d63a05b6f11106858543b32a385110afd942e91209513280595" Oct 30 05:31:46.982181 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:31:46.982161 5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-wr9wr_openshift-insights(60d71bc3-a236-4305-baeb-df6403b52f43)\"" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" podUID="60d71bc3-a236-4305-baeb-df6403b52f43" Oct 30 05:31:46.983497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.983478 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869865c857-td99r" Oct 30 05:31:46.983598 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:46.983480 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869865c857-td99r" event={"ID":"2fd827f0-3153-4b94-8326-8c6fd0744f6b","Type":"ContainerDied","Data":"191a33a9a7eef4281403d2e80939e756d127c3b26f344e3cdd493ef02c200adb"} Oct 30 05:31:47.027710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:47.027513 5072 scope.go:122] "RemoveContainer" containerID="c200e18e14849ed2120440d3f15082549d522eadbad0a539d1ac32c5f38c5db2" Oct 30 05:31:47.065724 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:47.065692 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-869865c857-td99r"] Oct 30 05:31:47.069338 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:47.069313 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-869865c857-td99r"] Oct 30 05:31:47.975936 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:47.975891 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-845d4d8d7d-6df2c" podUID="d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" containerName="console" containerID="cri-o://9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4" gracePeriod=15 Oct 30 05:31:48.249736 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.249663 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:31:48.277536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.277501 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-service-ca\") pod \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " Oct 30 05:31:48.277667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.277559 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-serving-cert\") pod \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " Oct 30 05:31:48.277667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.277600 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-kube-api-access-wlwts\" (UniqueName: \"kubernetes.io/projected/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-kube-api-access-wlwts\") pod \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " Oct 30 05:31:48.277667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.277628 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-oauth-serving-cert\") pod \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " Oct 30 05:31:48.277667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.277662 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-trusted-ca-bundle\") pod \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " Oct 30 05:31:48.277947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.277682 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-oauth-config\") pod \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " Oct 30 05:31:48.277947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.277714 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-tmp\") pod \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " Oct 30 05:31:48.277947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.277784 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-config\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-config\") pod \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\" (UID: \"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f\") " Oct 30 05:31:48.277947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.277901 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-service-ca" pod "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" (UID: "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:48.278153 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.278012 5072 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.278153 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.278118 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-tmp" pod "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" (UID: "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:48.278240 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.278134 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-oauth-serving-cert" pod "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" (UID: "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:48.278302 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.278283 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-config" pod "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" (UID: "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:48.278351 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.278312 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-trusted-ca-bundle" pod "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" (UID: "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:48.280154 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.280133 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-oauth-config" pod "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" (UID: "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:48.280501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.280473 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-serving-cert" pod "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" (UID: "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:48.280501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.280477 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-kube-api-access-wlwts" pod "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" (UID: "d9f4df7d-3912-4d92-a0fb-83e936b8ff9f"). InnerVolumeSpecName "kube-api-access-wlwts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:48.378792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.378759 5072 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.378792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.378787 5072 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.378792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.378798 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlwts\" (UniqueName: \"kubernetes.io/projected/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-kube-api-access-wlwts\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.379012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.378808 5072 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-oauth-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.379012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.378818 5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-trusted-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.379012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.378826 5072 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-console-oauth-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.379012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.378834 5072 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f-tmp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.517539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.517451 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd827f0-3153-4b94-8326-8c6fd0744f6b" path="/var/lib/kubelet/pods/2fd827f0-3153-4b94-8326-8c6fd0744f6b/volumes" Oct 30 05:31:48.996502 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.996445 5072 generic.go:356] "Generic (PLEG): container finished" podID="d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" containerID="9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4" exitCode=0 Oct 30 05:31:48.996749 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.996508 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845d4d8d7d-6df2c" event={"ID":"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f","Type":"ContainerDied","Data":"9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4"} Oct 30 05:31:48.996749 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.996535 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845d4d8d7d-6df2c" Oct 30 05:31:48.996749 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.996621 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845d4d8d7d-6df2c" event={"ID":"d9f4df7d-3912-4d92-a0fb-83e936b8ff9f","Type":"ContainerDied","Data":"702ffc54c45e8f19b65dbdc7225ea81c998843da46f36aa801a3c2c7d9104bac"} Oct 30 05:31:48.996749 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:48.996653 5072 scope.go:122] "RemoveContainer" containerID="9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4" Oct 30 05:31:49.030009 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:49.029985 5072 scope.go:122] "RemoveContainer" containerID="9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4" Oct 30 05:31:49.030324 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:31:49.030304 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4\": container with ID starting with 9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4 not found: ID does not exist" containerID="9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4" Oct 30 05:31:49.030399 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:49.030334 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4"} err="failed to get container status \"9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4\": rpc error: code = NotFound desc = could not find container \"9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4\": container with ID starting with 9a3117df69bfea523b10bd03bfb0fa49c2f4816fc84471e922bad1f68a7152a4 not found: ID does not exist" Oct 30 05:31:49.048511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:49.048484 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845d4d8d7d-6df2c"] Oct 30 05:31:49.060883 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:49.060859 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-845d4d8d7d-6df2c"] Oct 30 05:31:50.517035 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:50.516993 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" path="/var/lib/kubelet/pods/d9f4df7d-3912-4d92-a0fb-83e936b8ff9f/volumes" Oct 30 05:31:54.949188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949153 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-845974d575-twr9q"] Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949476 5072 cpu_manager.go:414] "containerMap: removing container" podUID="35863a55-5422-4304-ad71-afddab4d4807" containerName="registry" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949487 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35863a55-5422-4304-ad71-afddab4d4807" containerName="registry" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949503 5072 cpu_manager.go:414] "containerMap: removing container" podUID="50e77776-459b-434f-bc0a-49f3afce15e9" containerName="console" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949508 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e77776-459b-434f-bc0a-49f3afce15e9" containerName="console" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949517 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" containerName="console" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949521 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" containerName="console" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949535 5072 cpu_manager.go:414] "containerMap: removing container" podUID="2fd827f0-3153-4b94-8326-8c6fd0744f6b" containerName="registry" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949540 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd827f0-3153-4b94-8326-8c6fd0744f6b" containerName="registry" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949611 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e77776-459b-434f-bc0a-49f3afce15e9" containerName="console" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949618 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f4df7d-3912-4d92-a0fb-83e936b8ff9f" containerName="console" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949625 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd827f0-3153-4b94-8326-8c6fd0744f6b" containerName="registry" Oct 30 05:31:54.949647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.949633 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="35863a55-5422-4304-ad71-afddab4d4807" containerName="registry" Oct 30 05:31:54.980404 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.980377 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845974d575-twr9q"] Oct 30 05:31:54.980602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:54.980500 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.034830 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.034803 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-serving-cert\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.034830 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.034832 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-service-ca\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.035055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.034848 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c61673a-a2a6-4b83-b365-baa772886cce-tmp\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.035055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.034877 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rmg\" (UniqueName: \"kubernetes.io/projected/5c61673a-a2a6-4b83-b365-baa772886cce-kube-api-access-w8rmg\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.035055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.034934 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-oauth-config\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.035055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.034987 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-console-config\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.035055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.035042 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-trusted-ca-bundle\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.035055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.035056 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-oauth-serving-cert\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.135867 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.135834 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-trusted-ca-bundle\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.135867 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.135868 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-oauth-serving-cert\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.136083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.135896 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-serving-cert\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.136083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.135919 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-service-ca\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.136083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.135943 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c61673a-a2a6-4b83-b365-baa772886cce-tmp\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.136083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.135968 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rmg\" (UniqueName: \"kubernetes.io/projected/5c61673a-a2a6-4b83-b365-baa772886cce-kube-api-access-w8rmg\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.136083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.135999 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-oauth-config\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.136083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.136023 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-console-config\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.136385 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.136362 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c61673a-a2a6-4b83-b365-baa772886cce-tmp\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.136839 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.136798 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-oauth-serving-cert\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.136839 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.136811 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-console-config\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.137189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.136836 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-service-ca\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.137189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.136989 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-trusted-ca-bundle\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.138622 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.138595 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-oauth-config\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.138777 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.138760 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-serving-cert\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.159786 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.159763 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rmg\" (UniqueName: \"kubernetes.io/projected/5c61673a-a2a6-4b83-b365-baa772886cce-kube-api-access-w8rmg\") pod \"console-845974d575-twr9q\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.315439 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.315360 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845974d575-twr9q" Oct 30 05:31:55.645572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:55.645541 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845974d575-twr9q"] Oct 30 05:31:55.646640 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:31:55.646614 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c61673a_a2a6_4b83_b365_baa772886cce.slice/crio-3d8faccd6381d7e6e7b97277ba5442f82866ac17ef4e5b7f317643d17595c84b WatchSource:0}: Error finding container 3d8faccd6381d7e6e7b97277ba5442f82866ac17ef4e5b7f317643d17595c84b: Status 404 returned error can't find the container with id 3d8faccd6381d7e6e7b97277ba5442f82866ac17ef4e5b7f317643d17595c84b Oct 30 05:31:56.028236 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:56.028155 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845974d575-twr9q" event={"ID":"5c61673a-a2a6-4b83-b365-baa772886cce","Type":"ContainerStarted","Data":"2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3"} Oct 30 05:31:56.028236 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:56.028216 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845974d575-twr9q" event={"ID":"5c61673a-a2a6-4b83-b365-baa772886cce","Type":"ContainerStarted","Data":"3d8faccd6381d7e6e7b97277ba5442f82866ac17ef4e5b7f317643d17595c84b"} Oct 30 05:31:56.073180 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:56.073122 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-845974d575-twr9q" podStartSLOduration=2.073110026 podStartE2EDuration="2.073110026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:56.072132254 +0000 UTC m=+75.675821522" watchObservedRunningTime="2026-10-30 05:31:56.073110026 +0000 UTC m=+75.676799364" Oct 30 05:31:59.511222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:31:59.511194 5072 scope.go:122] "RemoveContainer" containerID="158451f32fc13d63a05b6f11106858543b32a385110afd942e91209513280595" Oct 30 05:32:00.047128 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.047092 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wr9wr" event={"ID":"60d71bc3-a236-4305-baeb-df6403b52f43","Type":"ContainerStarted","Data":"f6be97a7fc1b6523200fb1710469f5a0d021ef0ca73ec83dad0df09cef160319"} Oct 30 05:32:00.210129 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.210094 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-6hlgb-qtlr7"] Oct 30 05:32:00.210305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.210286 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" podUID="b9d914f2-a350-49db-b937-c4774be4415e" containerName="insights-gathering" containerID="cri-o://fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d" gracePeriod=30 Oct 30 05:32:00.495661 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.495636 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:32:00.583340 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.583302 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle\") pod \"b9d914f2-a350-49db-b937-c4774be4415e\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " Oct 30 05:32:00.583821 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.583359 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle\") pod \"b9d914f2-a350-49db-b937-c4774be4415e\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " Oct 30 05:32:00.583821 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.583392 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz\" (UniqueName: \"kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz\") pod \"b9d914f2-a350-49db-b937-c4774be4415e\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " Oct 30 05:32:00.583821 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.583428 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path\") pod \"b9d914f2-a350-49db-b937-c4774be4415e\" (UID: \"b9d914f2-a350-49db-b937-c4774be4415e\") " Oct 30 05:32:00.583949 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.583828 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle" pod "b9d914f2-a350-49db-b937-c4774be4415e" (UID: "b9d914f2-a350-49db-b937-c4774be4415e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:00.583949 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.583835 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle" pod "b9d914f2-a350-49db-b937-c4774be4415e" (UID: "b9d914f2-a350-49db-b937-c4774be4415e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:00.583949 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.583840 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path" pod "b9d914f2-a350-49db-b937-c4774be4415e" (UID: "b9d914f2-a350-49db-b937-c4774be4415e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:00.586003 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.585977 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz" pod "b9d914f2-a350-49db-b937-c4774be4415e" (UID: "b9d914f2-a350-49db-b937-c4774be4415e"). InnerVolumeSpecName "kube-api-access-5kstz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:00.684527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.684498 5072 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-service-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:00.684527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.684526 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kstz\" (UniqueName: \"kubernetes.io/projected/b9d914f2-a350-49db-b937-c4774be4415e-kube-api-access-5kstz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:00.684743 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.684539 5072 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b9d914f2-a350-49db-b937-c4774be4415e-archives-path\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:00.684743 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:00.684552 5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d914f2-a350-49db-b937-c4774be4415e-trusted-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:00.707848 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:32:00.707816 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:32:01.054878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:01.054796 5072 generic.go:356] "Generic (PLEG): container finished" podID="b9d914f2-a350-49db-b937-c4774be4415e" containerID="fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d" exitCode=2 Oct 30 05:32:01.054878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:01.054849 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" Oct 30 05:32:01.054878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:01.054862 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" event={"ID":"b9d914f2-a350-49db-b937-c4774be4415e","Type":"ContainerDied","Data":"fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d"} Oct 30 05:32:01.055117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:01.054909 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6hlgb-qtlr7" event={"ID":"b9d914f2-a350-49db-b937-c4774be4415e","Type":"ContainerDied","Data":"4c76d6cc77a310ed1edf9da56a5c4a3742d5f51753860eabe60dc31345899e48"} Oct 30 05:32:01.055117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:01.054930 5072 scope.go:122] "RemoveContainer" containerID="fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d" Oct 30 05:32:01.063688 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:01.063663 5072 scope.go:122] "RemoveContainer" containerID="fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d" Oct 30 05:32:01.063953 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:32:01.063934 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d\": container with ID starting with fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d not found: ID does not exist" containerID="fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d" Oct 30 05:32:01.064012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:01.063964 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d"} err="failed to get container status \"fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d\": rpc error: code = NotFound desc = could not find container \"fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d\": container with ID starting with fdb938da0338d288327954032ddebe8411c8e5a8be3d8b280e4efae43b00d03d not found: ID does not exist" Oct 30 05:32:01.105935 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:01.105907 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-6hlgb-qtlr7"] Oct 30 05:32:01.119165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:01.119136 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-6hlgb-qtlr7"] Oct 30 05:32:02.515418 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:02.515382 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d914f2-a350-49db-b937-c4774be4415e" path="/var/lib/kubelet/pods/b9d914f2-a350-49db-b937-c4774be4415e/volumes" Oct 30 05:32:05.316023 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:05.315986 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-845974d575-twr9q" Oct 30 05:32:05.316540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:05.316038 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-845974d575-twr9q" Oct 30 05:32:05.320976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:05.320953 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-845974d575-twr9q" Oct 30 05:32:06.080563 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:06.080537 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-845974d575-twr9q" Oct 30 05:32:06.154392 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:06.154356 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5445c766fd-qj666"] Oct 30 05:32:30.708863 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:32:30.708823 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:32:31.178304 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.178266 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5445c766fd-qj666" podUID="91d12839-c1e9-481d-9b3e-65065606dd3d" containerName="console" containerID="cri-o://1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7" gracePeriod=15 Oct 30 05:32:31.436738 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.436683 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:32:31.547702 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.547664 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-trusted-ca-bundle\") pod \"91d12839-c1e9-481d-9b3e-65065606dd3d\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " Oct 30 05:32:31.547702 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.547701 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-console-config\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-console-config\") pod \"91d12839-c1e9-481d-9b3e-65065606dd3d\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " Oct 30 05:32:31.547958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.547727 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-oauth-config\") pod \"91d12839-c1e9-481d-9b3e-65065606dd3d\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " Oct 30 05:32:31.547958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.547777 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-serving-cert\") pod \"91d12839-c1e9-481d-9b3e-65065606dd3d\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " Oct 30 05:32:31.547958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.547806 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-oauth-serving-cert\") pod \"91d12839-c1e9-481d-9b3e-65065606dd3d\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " Oct 30 05:32:31.547958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.547837 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-service-ca\") pod \"91d12839-c1e9-481d-9b3e-65065606dd3d\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " Oct 30 05:32:31.547958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.547872 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d12839-c1e9-481d-9b3e-65065606dd3d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d12839-c1e9-481d-9b3e-65065606dd3d-tmp\") pod \"91d12839-c1e9-481d-9b3e-65065606dd3d\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " Oct 30 05:32:31.547958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.547899 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d12839-c1e9-481d-9b3e-65065606dd3d-kube-api-access-k7vf7\" (UniqueName: \"kubernetes.io/projected/91d12839-c1e9-481d-9b3e-65065606dd3d-kube-api-access-k7vf7\") pod \"91d12839-c1e9-481d-9b3e-65065606dd3d\" (UID: \"91d12839-c1e9-481d-9b3e-65065606dd3d\") " Oct 30 05:32:31.548349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.548207 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d12839-c1e9-481d-9b3e-65065606dd3d-tmp" pod "91d12839-c1e9-481d-9b3e-65065606dd3d" (UID: "91d12839-c1e9-481d-9b3e-65065606dd3d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:31.548349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.548211 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-console-config" pod "91d12839-c1e9-481d-9b3e-65065606dd3d" (UID: "91d12839-c1e9-481d-9b3e-65065606dd3d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:31.548349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.548258 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-trusted-ca-bundle" pod "91d12839-c1e9-481d-9b3e-65065606dd3d" (UID: "91d12839-c1e9-481d-9b3e-65065606dd3d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:31.548522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.548446 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-oauth-serving-cert" pod "91d12839-c1e9-481d-9b3e-65065606dd3d" (UID: "91d12839-c1e9-481d-9b3e-65065606dd3d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:31.548584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.548528 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-service-ca" pod "91d12839-c1e9-481d-9b3e-65065606dd3d" (UID: "91d12839-c1e9-481d-9b3e-65065606dd3d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:31.550323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.550291 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-oauth-config" pod "91d12839-c1e9-481d-9b3e-65065606dd3d" (UID: "91d12839-c1e9-481d-9b3e-65065606dd3d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:31.550323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.550307 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-serving-cert" pod "91d12839-c1e9-481d-9b3e-65065606dd3d" (UID: "91d12839-c1e9-481d-9b3e-65065606dd3d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:31.550489 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.550402 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d12839-c1e9-481d-9b3e-65065606dd3d-kube-api-access-k7vf7" pod "91d12839-c1e9-481d-9b3e-65065606dd3d" (UID: "91d12839-c1e9-481d-9b3e-65065606dd3d"). InnerVolumeSpecName "kube-api-access-k7vf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:31.648712 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.648678 5072 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.648712 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.648705 5072 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-oauth-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.648712 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.648714 5072 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.648921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.648723 5072 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d12839-c1e9-481d-9b3e-65065606dd3d-tmp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.648921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.648731 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7vf7\" (UniqueName: \"kubernetes.io/projected/91d12839-c1e9-481d-9b3e-65065606dd3d-kube-api-access-k7vf7\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.648921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.648740 5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-trusted-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.648921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.648748 5072 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91d12839-c1e9-481d-9b3e-65065606dd3d-console-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:31.648921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:31.648756 5072 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91d12839-c1e9-481d-9b3e-65065606dd3d-console-oauth-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:32:32.179829 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.179737 5072 generic.go:356] "Generic (PLEG): container finished" podID="91d12839-c1e9-481d-9b3e-65065606dd3d" containerID="1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7" exitCode=0 Oct 30 05:32:32.179829 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.179782 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5445c766fd-qj666" Oct 30 05:32:32.179829 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.179817 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5445c766fd-qj666" event={"ID":"91d12839-c1e9-481d-9b3e-65065606dd3d","Type":"ContainerDied","Data":"1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7"} Oct 30 05:32:32.180323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.179868 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5445c766fd-qj666" event={"ID":"91d12839-c1e9-481d-9b3e-65065606dd3d","Type":"ContainerDied","Data":"94ece3bd92bab72db28e70eb144f2624cf40e932db21ba5e3abea13ed1caed47"} Oct 30 05:32:32.180323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.179890 5072 scope.go:122] "RemoveContainer" containerID="1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7" Oct 30 05:32:32.188643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.188629 5072 scope.go:122] "RemoveContainer" containerID="1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7" Oct 30 05:32:32.188900 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:32:32.188884 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7\": container with ID starting with 1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7 not found: ID does not exist" containerID="1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7" Oct 30 05:32:32.188951 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.188910 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7"} err="failed to get container status \"1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7\": rpc error: code = NotFound desc = could not find container \"1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7\": container with ID starting with 1c0ef9e934da4ab224616537da90f78ae905658577603905a306f3de7408dbf7 not found: ID does not exist" Oct 30 05:32:32.232124 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.232097 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5445c766fd-qj666"] Oct 30 05:32:32.235386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.235367 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5445c766fd-qj666"] Oct 30 05:32:32.515166 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:32.515090 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d12839-c1e9-481d-9b3e-65065606dd3d" path="/var/lib/kubelet/pods/91d12839-c1e9-481d-9b3e-65065606dd3d/volumes" Oct 30 05:32:42.658061 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.658026 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f64d85cc-jtdhf"] Oct 30 05:32:42.658608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.658375 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b9d914f2-a350-49db-b937-c4774be4415e" containerName="insights-gathering" Oct 30 05:32:42.658608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.658389 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d914f2-a350-49db-b937-c4774be4415e" containerName="insights-gathering" Oct 30 05:32:42.658608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.658408 5072 cpu_manager.go:414] "containerMap: removing container" podUID="91d12839-c1e9-481d-9b3e-65065606dd3d" containerName="console" Oct 30 05:32:42.658608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.658413 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d12839-c1e9-481d-9b3e-65065606dd3d" containerName="console" Oct 30 05:32:42.658608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.658496 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d12839-c1e9-481d-9b3e-65065606dd3d" containerName="console" Oct 30 05:32:42.658608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.658509 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d914f2-a350-49db-b937-c4774be4415e" containerName="insights-gathering" Oct 30 05:32:42.662899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.662882 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.687866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.687835 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f64d85cc-jtdhf"] Oct 30 05:32:42.738649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.738617 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-oauth-serving-cert\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.738649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.738646 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b2ec566-a1d7-41e5-9460-4a35b77da90e-console-serving-cert\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.738825 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.738667 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hk7j\" (UniqueName: \"kubernetes.io/projected/5b2ec566-a1d7-41e5-9460-4a35b77da90e-kube-api-access-6hk7j\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.738825 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.738684 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b2ec566-a1d7-41e5-9460-4a35b77da90e-console-oauth-config\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.738825 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.738748 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-service-ca\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.738825 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.738807 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-console-config\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.738825 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.738826 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b2ec566-a1d7-41e5-9460-4a35b77da90e-tmp\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.738978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.738865 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-trusted-ca-bundle\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.839477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.839427 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-oauth-serving-cert\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.839477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.839477 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b2ec566-a1d7-41e5-9460-4a35b77da90e-console-serving-cert\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.839713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.839501 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hk7j\" (UniqueName: \"kubernetes.io/projected/5b2ec566-a1d7-41e5-9460-4a35b77da90e-kube-api-access-6hk7j\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.839713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.839523 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b2ec566-a1d7-41e5-9460-4a35b77da90e-console-oauth-config\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.839713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.839545 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-service-ca\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.839713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.839576 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-console-config\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.839713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.839598 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b2ec566-a1d7-41e5-9460-4a35b77da90e-tmp\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.839713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.839666 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-trusted-ca-bundle\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.840300 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.840272 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-oauth-serving-cert\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.840432 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.840327 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b2ec566-a1d7-41e5-9460-4a35b77da90e-tmp\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.840432 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.840344 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-service-ca\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.840662 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.840635 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-console-config\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.840978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.840958 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b2ec566-a1d7-41e5-9460-4a35b77da90e-trusted-ca-bundle\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.842368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.842344 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b2ec566-a1d7-41e5-9460-4a35b77da90e-console-oauth-config\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.842497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.842431 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b2ec566-a1d7-41e5-9460-4a35b77da90e-console-serving-cert\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.863692 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.863669 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hk7j\" (UniqueName: \"kubernetes.io/projected/5b2ec566-a1d7-41e5-9460-4a35b77da90e-kube-api-access-6hk7j\") pod \"console-67f64d85cc-jtdhf\" (UID: \"5b2ec566-a1d7-41e5-9460-4a35b77da90e\") " pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:42.975181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:42.975096 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:43.272893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:43.272867 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f64d85cc-jtdhf"] Oct 30 05:32:43.273606 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:32:43.273584 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2ec566_a1d7_41e5_9460_4a35b77da90e.slice/crio-1ec313208c2078f4c8632fbd6d3874d30afdd1343abf2246cf00c37b4153ed58 WatchSource:0}: Error finding container 1ec313208c2078f4c8632fbd6d3874d30afdd1343abf2246cf00c37b4153ed58: Status 404 returned error can't find the container with id 1ec313208c2078f4c8632fbd6d3874d30afdd1343abf2246cf00c37b4153ed58 Oct 30 05:32:44.232118 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:44.232081 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f64d85cc-jtdhf" event={"ID":"5b2ec566-a1d7-41e5-9460-4a35b77da90e","Type":"ContainerStarted","Data":"38913988651bf9ff2e75c254129823db598bea7f6e7f48be1636ec2d47e8813e"} Oct 30 05:32:44.232545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:44.232131 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f64d85cc-jtdhf" event={"ID":"5b2ec566-a1d7-41e5-9460-4a35b77da90e","Type":"ContainerStarted","Data":"1ec313208c2078f4c8632fbd6d3874d30afdd1343abf2246cf00c37b4153ed58"} Oct 30 05:32:44.276436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:44.276389 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67f64d85cc-jtdhf" podStartSLOduration=2.276377803 podStartE2EDuration="2.276377803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:44.276144853 +0000 UTC m=+123.879834109" watchObservedRunningTime="2026-10-30 05:32:44.276377803 +0000 UTC m=+123.880067054" Oct 30 05:32:52.976289 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:52.976190 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:52.976890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:52.976302 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:52.981172 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:52.981146 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:53.274020 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:53.273946 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f64d85cc-jtdhf" Oct 30 05:32:53.363145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:32:53.363112 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845974d575-twr9q"] Oct 30 05:33:00.708293 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:33:00.708258 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:33:18.385812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.385751 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-845974d575-twr9q" podUID="5c61673a-a2a6-4b83-b365-baa772886cce" containerName="console" containerID="cri-o://2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3" gracePeriod=15 Oct 30 05:33:18.632944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.632919 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845974d575-twr9q" Oct 30 05:33:18.746024 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.745933 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c61673a-a2a6-4b83-b365-baa772886cce-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c61673a-a2a6-4b83-b365-baa772886cce-tmp\") pod \"5c61673a-a2a6-4b83-b365-baa772886cce\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " Oct 30 05:33:18.746024 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.745979 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-console-config\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-console-config\") pod \"5c61673a-a2a6-4b83-b365-baa772886cce\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " Oct 30 05:33:18.746024 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746003 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-oauth-config\") pod \"5c61673a-a2a6-4b83-b365-baa772886cce\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " Oct 30 05:33:18.746319 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746038 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-oauth-serving-cert\") pod \"5c61673a-a2a6-4b83-b365-baa772886cce\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " Oct 30 05:33:18.746319 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746067 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c61673a-a2a6-4b83-b365-baa772886cce-kube-api-access-w8rmg\" (UniqueName: \"kubernetes.io/projected/5c61673a-a2a6-4b83-b365-baa772886cce-kube-api-access-w8rmg\") pod \"5c61673a-a2a6-4b83-b365-baa772886cce\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " Oct 30 05:33:18.746319 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746108 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-serving-cert\") pod \"5c61673a-a2a6-4b83-b365-baa772886cce\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " Oct 30 05:33:18.746319 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746135 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-trusted-ca-bundle\") pod \"5c61673a-a2a6-4b83-b365-baa772886cce\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " Oct 30 05:33:18.746319 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746211 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-service-ca\") pod \"5c61673a-a2a6-4b83-b365-baa772886cce\" (UID: \"5c61673a-a2a6-4b83-b365-baa772886cce\") " Oct 30 05:33:18.746319 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746278 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c61673a-a2a6-4b83-b365-baa772886cce-tmp" pod "5c61673a-a2a6-4b83-b365-baa772886cce" (UID: "5c61673a-a2a6-4b83-b365-baa772886cce"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:33:18.746635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746541 5072 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c61673a-a2a6-4b83-b365-baa772886cce-tmp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:33:18.746635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746549 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-console-config" pod "5c61673a-a2a6-4b83-b365-baa772886cce" (UID: "5c61673a-a2a6-4b83-b365-baa772886cce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:18.746635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746610 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-oauth-serving-cert" pod "5c61673a-a2a6-4b83-b365-baa772886cce" (UID: "5c61673a-a2a6-4b83-b365-baa772886cce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:18.746771 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746669 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-service-ca" pod "5c61673a-a2a6-4b83-b365-baa772886cce" (UID: "5c61673a-a2a6-4b83-b365-baa772886cce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:18.746771 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.746710 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-trusted-ca-bundle" pod "5c61673a-a2a6-4b83-b365-baa772886cce" (UID: "5c61673a-a2a6-4b83-b365-baa772886cce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:18.748395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.748371 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-oauth-config" pod "5c61673a-a2a6-4b83-b365-baa772886cce" (UID: "5c61673a-a2a6-4b83-b365-baa772886cce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:18.748646 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.748627 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-serving-cert" pod "5c61673a-a2a6-4b83-b365-baa772886cce" (UID: "5c61673a-a2a6-4b83-b365-baa772886cce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:18.748705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.748648 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c61673a-a2a6-4b83-b365-baa772886cce-kube-api-access-w8rmg" pod "5c61673a-a2a6-4b83-b365-baa772886cce" (UID: "5c61673a-a2a6-4b83-b365-baa772886cce"). InnerVolumeSpecName "kube-api-access-w8rmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:18.847960 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.847925 5072 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-console-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:33:18.847960 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.847953 5072 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-oauth-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:33:18.847960 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.847963 5072 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-oauth-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:33:18.848197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.847972 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8rmg\" (UniqueName: \"kubernetes.io/projected/5c61673a-a2a6-4b83-b365-baa772886cce-kube-api-access-w8rmg\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:33:18.848197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.847983 5072 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61673a-a2a6-4b83-b365-baa772886cce-console-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:33:18.848197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.847993 5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-trusted-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:33:18.848197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:18.848003 5072 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61673a-a2a6-4b83-b365-baa772886cce-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:33:19.377199 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:19.377162 5072 generic.go:356] "Generic (PLEG): container finished" podID="5c61673a-a2a6-4b83-b365-baa772886cce" containerID="2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3" exitCode=0 Oct 30 05:33:19.377394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:19.377207 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845974d575-twr9q" event={"ID":"5c61673a-a2a6-4b83-b365-baa772886cce","Type":"ContainerDied","Data":"2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3"} Oct 30 05:33:19.377394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:19.377217 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845974d575-twr9q" Oct 30 05:33:19.377394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:19.377248 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845974d575-twr9q" event={"ID":"5c61673a-a2a6-4b83-b365-baa772886cce","Type":"ContainerDied","Data":"3d8faccd6381d7e6e7b97277ba5442f82866ac17ef4e5b7f317643d17595c84b"} Oct 30 05:33:19.377394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:19.377267 5072 scope.go:122] "RemoveContainer" containerID="2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3" Oct 30 05:33:19.386006 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:19.385990 5072 scope.go:122] "RemoveContainer" containerID="2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3" Oct 30 05:33:19.386334 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:33:19.386300 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3\": container with ID starting with 2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3 not found: ID does not exist" containerID="2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3" Oct 30 05:33:19.386379 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:19.386329 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3"} err="failed to get container status \"2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3\": rpc error: code = NotFound desc = could not find container \"2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3\": container with ID starting with 2b724ee72a617c2ba5de3e8b826852cb24b1141ad9a37206338c3019e994afd3 not found: ID does not exist" Oct 30 05:33:19.428737 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:19.428708 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845974d575-twr9q"] Oct 30 05:33:19.441063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:19.441037 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-845974d575-twr9q"] Oct 30 05:33:20.515472 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:33:20.515425 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c61673a-a2a6-4b83-b365-baa772886cce" path="/var/lib/kubelet/pods/5c61673a-a2a6-4b83-b365-baa772886cce/volumes" Oct 30 05:33:30.708356 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:33:30.708321 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:34:00.707566 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:34:00.707532 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:34:30.708441 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:34:30.708396 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:35:00.708285 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:35:00.708252 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:35:30.709603 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:35:30.709566 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:35:40.527243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:35:40.527213 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:35:40.531746 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:35:40.531724 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:36:00.709900 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:36:00.709814 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:36:30.710824 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:36:30.710789 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:36:34.082541 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.082504 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8"] Oct 30 05:36:34.083194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.082856 5072 cpu_manager.go:414] "containerMap: removing container" podUID="5c61673a-a2a6-4b83-b365-baa772886cce" containerName="console" Oct 30 05:36:34.083194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.082866 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c61673a-a2a6-4b83-b365-baa772886cce" containerName="console" Oct 30 05:36:34.083194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.082938 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c61673a-a2a6-4b83-b365-baa772886cce" containerName="console" Oct 30 05:36:34.086274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.086253 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.110891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.110868 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 05:36:34.111048 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.110923 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 05:36:34.111048 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.110929 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 05:36:34.111314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.111235 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 05:36:34.111314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.111261 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wrhw7\"" Oct 30 05:36:34.111550 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.111525 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 05:36:34.123401 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.123377 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8"] Oct 30 05:36:34.223374 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.223339 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.223576 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.223393 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-manifests\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.223576 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.223417 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.223576 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.223447 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-tmp\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.223719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.223592 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2jk\" (UniqueName: \"kubernetes.io/projected/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-kube-api-access-2b2jk\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.223719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.223628 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.324922 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.324888 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.325109 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.324937 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-manifests\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.325109 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.324956 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.325109 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.324971 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-tmp\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.325109 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.325021 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2jk\" (UniqueName: \"kubernetes.io/projected/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-kube-api-access-2b2jk\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.325109 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.325041 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.325388 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.325370 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-manifests\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.327590 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.327563 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-tmp\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.327773 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.327750 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.327819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.327791 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.327819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.327808 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.347135 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.347080 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2jk\" (UniqueName: \"kubernetes.io/projected/ed7b64b6-6071-4376-aaf5-38a2baa4c70f-kube-api-access-2b2jk\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-whrp8\" (UID: \"ed7b64b6-6071-4376-aaf5-38a2baa4c70f\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.401175 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.401148 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:34.751239 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.751202 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8"] Oct 30 05:36:34.764670 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:36:34.764643 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7b64b6_6071_4376_aaf5_38a2baa4c70f.slice/crio-741bc1ea2e96e142b28551217504f259eaca3444f05c97e5e09f94e27ea2cee8 WatchSource:0}: Error finding container 741bc1ea2e96e142b28551217504f259eaca3444f05c97e5e09f94e27ea2cee8: Status 404 returned error can't find the container with id 741bc1ea2e96e142b28551217504f259eaca3444f05c97e5e09f94e27ea2cee8 Oct 30 05:36:34.766201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:34.766185 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:36:35.151173 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:35.151135 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" event={"ID":"ed7b64b6-6071-4376-aaf5-38a2baa4c70f","Type":"ContainerStarted","Data":"741bc1ea2e96e142b28551217504f259eaca3444f05c97e5e09f94e27ea2cee8"} Oct 30 05:36:38.167724 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:38.167690 5072 generic.go:356] "Generic (PLEG): container finished" podID="ed7b64b6-6071-4376-aaf5-38a2baa4c70f" containerID="79eacffbd2cff452032cb7726370e9f9254aab5511b5d1e27a34e5560bd01cb6" exitCode=0 Oct 30 05:36:38.168104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:38.167776 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" event={"ID":"ed7b64b6-6071-4376-aaf5-38a2baa4c70f","Type":"ContainerDied","Data":"79eacffbd2cff452032cb7726370e9f9254aab5511b5d1e27a34e5560bd01cb6"} Oct 30 05:36:39.174093 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:39.174054 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" event={"ID":"ed7b64b6-6071-4376-aaf5-38a2baa4c70f","Type":"ContainerStarted","Data":"49de2010a232ae3ca36ea5ed205fbcdfa8e609eae828290c46c949c82051e9a0"} Oct 30 05:36:39.174517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:39.174222 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:39.268225 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:39.268156 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" podStartSLOduration=2.716633162 podStartE2EDuration="6.26813628s" totalImagesPullingTime="3.1490312s" totalInitContainerRuntime="402.471918ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:39.256890069 +0000 UTC m=+358.860579326" watchObservedRunningTime="2026-10-30 05:36:39.26813628 +0000 UTC m=+358.871825551" Oct 30 05:36:50.185843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:50.185815 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-whrp8" Oct 30 05:36:55.859890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:55.859853 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-788bd9946-g8t5n"] Oct 30 05:36:55.863924 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:55.863892 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:55.878448 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:55.878411 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 05:36:55.878604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:55.878531 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 05:36:55.889291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:55.889266 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-22vcj\"" Oct 30 05:36:55.891005 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:55.890984 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788bd9946-g8t5n"] Oct 30 05:36:55.905756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:55.905734 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14e11f60-a1b9-4a63-bb32-60f3388808f8-tmp\") pod \"kube-auth-proxy-788bd9946-g8t5n\" (UID: \"14e11f60-a1b9-4a63-bb32-60f3388808f8\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:55.905867 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:55.905775 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e11f60-a1b9-4a63-bb32-60f3388808f8-tls-certs\") pod \"kube-auth-proxy-788bd9946-g8t5n\" (UID: \"14e11f60-a1b9-4a63-bb32-60f3388808f8\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:55.905867 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:55.905805 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7sf\" (UniqueName: \"kubernetes.io/projected/14e11f60-a1b9-4a63-bb32-60f3388808f8-kube-api-access-nw7sf\") pod \"kube-auth-proxy-788bd9946-g8t5n\" (UID: \"14e11f60-a1b9-4a63-bb32-60f3388808f8\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:56.006957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:56.006923 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e11f60-a1b9-4a63-bb32-60f3388808f8-tls-certs\") pod \"kube-auth-proxy-788bd9946-g8t5n\" (UID: \"14e11f60-a1b9-4a63-bb32-60f3388808f8\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:56.006957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:56.006962 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7sf\" (UniqueName: \"kubernetes.io/projected/14e11f60-a1b9-4a63-bb32-60f3388808f8-kube-api-access-nw7sf\") pod \"kube-auth-proxy-788bd9946-g8t5n\" (UID: \"14e11f60-a1b9-4a63-bb32-60f3388808f8\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:56.007215 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:56.007032 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14e11f60-a1b9-4a63-bb32-60f3388808f8-tmp\") pod \"kube-auth-proxy-788bd9946-g8t5n\" (UID: \"14e11f60-a1b9-4a63-bb32-60f3388808f8\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:56.009452 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:56.009416 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14e11f60-a1b9-4a63-bb32-60f3388808f8-tmp\") pod \"kube-auth-proxy-788bd9946-g8t5n\" (UID: \"14e11f60-a1b9-4a63-bb32-60f3388808f8\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:56.009598 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:56.009543 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e11f60-a1b9-4a63-bb32-60f3388808f8-tls-certs\") pod \"kube-auth-proxy-788bd9946-g8t5n\" (UID: \"14e11f60-a1b9-4a63-bb32-60f3388808f8\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:56.030322 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:56.030300 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7sf\" (UniqueName: \"kubernetes.io/projected/14e11f60-a1b9-4a63-bb32-60f3388808f8-kube-api-access-nw7sf\") pod \"kube-auth-proxy-788bd9946-g8t5n\" (UID: \"14e11f60-a1b9-4a63-bb32-60f3388808f8\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:56.176016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:56.175927 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" Oct 30 05:36:56.472749 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:56.472665 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788bd9946-g8t5n"] Oct 30 05:36:56.475662 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:36:56.475637 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e11f60_a1b9_4a63_bb32_60f3388808f8.slice/crio-0404352894a23b6894b8e291e809ee61b1e7a931482aa54e957c9e9d68426d51 WatchSource:0}: Error finding container 0404352894a23b6894b8e291e809ee61b1e7a931482aa54e957c9e9d68426d51: Status 404 returned error can't find the container with id 0404352894a23b6894b8e291e809ee61b1e7a931482aa54e957c9e9d68426d51 Oct 30 05:36:57.258483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.258425 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" event={"ID":"14e11f60-a1b9-4a63-bb32-60f3388808f8","Type":"ContainerStarted","Data":"0404352894a23b6894b8e291e809ee61b1e7a931482aa54e957c9e9d68426d51"} Oct 30 05:36:57.475694 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.475649 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s"] Oct 30 05:36:57.479320 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.479298 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.493190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.493166 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 05:36:57.496082 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.496062 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 05:36:57.508602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.508533 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s"] Oct 30 05:36:57.520244 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.520202 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.520386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.520311 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.520386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.520347 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.520386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.520380 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.520555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.520451 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a64752-7a83-4f0b-a1fa-39175a625238-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.520555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.520494 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.520643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.520607 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqmr\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-kube-api-access-mnqmr\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.592681 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.592648 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 05:36:57.592681 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.592648 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 05:36:57.592878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.592691 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h6vpc\"" Oct 30 05:36:57.621550 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.621516 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.621718 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.621589 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.621718 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.621628 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.621718 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.621657 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.621718 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.621691 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a64752-7a83-4f0b-a1fa-39175a625238-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.621718 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.621707 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.621928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.621734 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqmr\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-kube-api-access-mnqmr\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.623797 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.623768 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.625068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.625042 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a64752-7a83-4f0b-a1fa-39175a625238-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.625512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.625492 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.625716 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.625693 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.629809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.629784 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.658491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.655498 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqmr\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-kube-api-access-mnqmr\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.658491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.655885 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wrg5s\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:57.796152 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:57.796068 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:36:58.134940 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:58.134911 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s"] Oct 30 05:36:58.135609 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:36:58.135574 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a64752_7a83_4f0b_a1fa_39175a625238.slice/crio-86d0752b0d3abbb95992d306a949f9756173c2ad6b5fc538536f7c21490e10f3 WatchSource:0}: Error finding container 86d0752b0d3abbb95992d306a949f9756173c2ad6b5fc538536f7c21490e10f3: Status 404 returned error can't find the container with id 86d0752b0d3abbb95992d306a949f9756173c2ad6b5fc538536f7c21490e10f3 Oct 30 05:36:58.265168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:58.265135 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" event={"ID":"d5a64752-7a83-4f0b-a1fa-39175a625238","Type":"ContainerStarted","Data":"86d0752b0d3abbb95992d306a949f9756173c2ad6b5fc538536f7c21490e10f3"} Oct 30 05:36:59.281899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:59.281812 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" event={"ID":"14e11f60-a1b9-4a63-bb32-60f3388808f8","Type":"ContainerStarted","Data":"4c5fa17b28f416d1d72c29da1708c9042cdb01786fb725175f986b93458a0689"} Oct 30 05:36:59.325331 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:36:59.324438 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-788bd9946-g8t5n" podStartSLOduration=1.851065003 podStartE2EDuration="4.324423238s" totalImagesPullingTime="2.473358235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:59.323817322 +0000 UTC m=+378.927506772" watchObservedRunningTime="2026-10-30 05:36:59.324423238 +0000 UTC m=+378.928112494" Oct 30 05:37:00.710811 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:37:00.710777 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:37:00.899559 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:00.899530 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:37:00.899623 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:00.899593 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:37:01.293249 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:01.293208 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" event={"ID":"d5a64752-7a83-4f0b-a1fa-39175a625238","Type":"ContainerStarted","Data":"9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c"} Oct 30 05:37:01.293574 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:01.293556 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:37:01.295282 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:01.295254 5072 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-wrg5s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:37:01.295418 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:01.295306 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" podUID="d5a64752-7a83-4f0b-a1fa-39175a625238" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:37:01.357268 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:01.357213 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" podStartSLOduration=1.595865796 podStartE2EDuration="4.357199155s" totalImagesPullingTime="2.761333359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:01.345918905 +0000 UTC m=+380.949608163" watchObservedRunningTime="2026-10-30 05:37:01.357199155 +0000 UTC m=+380.960888413" Oct 30 05:37:02.299835 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:02.299803 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:37:07.889087 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:07.889047 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s"] Oct 30 05:37:07.893252 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:07.893236 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:07.906625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:07.906602 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-tr67c\"" Oct 30 05:37:07.939729 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:07.939685 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 05:37:07.941209 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:07.941190 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s"] Oct 30 05:37:08.018366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.018328 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvmv\" (UniqueName: \"kubernetes.io/projected/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-kube-api-access-xdvmv\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.018557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.018401 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.018557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.018499 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.018557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.018530 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.119745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.119709 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.119745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.119745 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.119966 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.119766 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.119966 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.119892 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvmv\" (UniqueName: \"kubernetes.io/projected/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-kube-api-access-xdvmv\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.120109 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.120089 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.120165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.120144 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.120589 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.120573 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.142766 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.142712 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvmv\" (UniqueName: \"kubernetes.io/projected/8fa1464f-f5fc-4013-b65e-9ad8ebd4188d-kube-api-access-xdvmv\") pod \"kserve-module-controller-manager-5cfb77bf8c-5rv7s\" (UID: \"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.205975 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.205950 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:08.517179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:08.517151 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s"] Oct 30 05:37:08.518007 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:37:08.517978 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa1464f_f5fc_4013_b65e_9ad8ebd4188d.slice/crio-82fb8be34a58416c0f3c184647ce1fb3b0a644e8b26a8e5367457f1c52251a84 WatchSource:0}: Error finding container 82fb8be34a58416c0f3c184647ce1fb3b0a644e8b26a8e5367457f1c52251a84: Status 404 returned error can't find the container with id 82fb8be34a58416c0f3c184647ce1fb3b0a644e8b26a8e5367457f1c52251a84 Oct 30 05:37:09.331613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:09.331570 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" event={"ID":"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d","Type":"ContainerStarted","Data":"82fb8be34a58416c0f3c184647ce1fb3b0a644e8b26a8e5367457f1c52251a84"} Oct 30 05:37:10.337045 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:10.336964 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" event={"ID":"8fa1464f-f5fc-4013-b65e-9ad8ebd4188d","Type":"ContainerStarted","Data":"880b52beaf76363f4fcbf3b427d59ac501fa5d2d73e5aabf618be12bef00372b"} Oct 30 05:37:10.337476 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:10.337100 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:10.390157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:10.390071 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" podStartSLOduration=1.902851388 podStartE2EDuration="3.390051108s" totalImagesPullingTime="1.48719972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:10.389531647 +0000 UTC m=+389.993220905" watchObservedRunningTime="2026-10-30 05:37:10.390051108 +0000 UTC m=+389.993740367" Oct 30 05:37:30.708706 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:37:30.708669 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:37:31.342587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:31.342547 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-5rv7s" Oct 30 05:37:55.230371 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.230336 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr"] Oct 30 05:37:55.235264 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.235244 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.266771 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.266729 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr"] Oct 30 05:37:55.338945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.338916 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.338945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.338951 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.339149 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.338981 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.339149 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.339059 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.339149 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.339091 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.339149 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.339114 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.339279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.339217 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68gp9\" (UniqueName: \"kubernetes.io/projected/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-kube-api-access-68gp9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.440407 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.440371 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gp9\" (UniqueName: \"kubernetes.io/projected/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-kube-api-access-68gp9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.440601 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.440432 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.440601 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.440478 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.440601 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.440522 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.440601 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.440549 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.440601 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.440572 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.440856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.440602 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.441580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.441552 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.443036 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.443013 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.443147 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.443093 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.443422 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.443400 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.443563 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.443553 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.462368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.462337 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.462500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.462406 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68gp9\" (UniqueName: \"kubernetes.io/projected/91613cf9-4b13-4b1a-ac74-90ec57a6eab5-kube-api-access-68gp9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-djzrr\" (UID: \"91613cf9-4b13-4b1a-ac74-90ec57a6eab5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.547386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.547305 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:55.892826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.890134 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr"] Oct 30 05:37:55.893769 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:37:55.893744 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91613cf9_4b13_4b1a_ac74_90ec57a6eab5.slice/crio-90fed919159132fe601052bbdb33a3f4a4dbb13d068fb9c35be51d44a69df4a0 WatchSource:0}: Error finding container 90fed919159132fe601052bbdb33a3f4a4dbb13d068fb9c35be51d44a69df4a0: Status 404 returned error can't find the container with id 90fed919159132fe601052bbdb33a3f4a4dbb13d068fb9c35be51d44a69df4a0 Oct 30 05:37:55.896393 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.896366 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:37:55.896478 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:55.896426 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:37:56.535155 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:56.535111 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" event={"ID":"91613cf9-4b13-4b1a-ac74-90ec57a6eab5","Type":"ContainerStarted","Data":"0af002c392ed573fc3a359a415b2bea70022859feea05e8ea6238b2c28782682"} Oct 30 05:37:56.535967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:56.535932 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" event={"ID":"91613cf9-4b13-4b1a-ac74-90ec57a6eab5","Type":"ContainerStarted","Data":"90fed919159132fe601052bbdb33a3f4a4dbb13d068fb9c35be51d44a69df4a0"} Oct 30 05:37:56.536060 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:56.535978 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:56.539053 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:56.539023 5072 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-djzrr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:37:56.539162 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:56.539084 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" podUID="91613cf9-4b13-4b1a-ac74-90ec57a6eab5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:37:56.604296 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:56.604216 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" podStartSLOduration=1.604198071 podStartE2EDuration="1.604198071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:56.602911946 +0000 UTC m=+436.206601206" watchObservedRunningTime="2026-10-30 05:37:56.604198071 +0000 UTC m=+436.207887328" Oct 30 05:37:57.540394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.540368 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-djzrr" Oct 30 05:37:57.634638 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.634603 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s"] Oct 30 05:37:57.634851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.634829 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" podUID="d5a64752-7a83-4f0b-a1fa-39175a625238" containerName="discovery" containerID="cri-o://9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c" gracePeriod=30 Oct 30 05:37:57.890386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.890358 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:37:57.964207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.964175 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-kube-api-access-mnqmr\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-kube-api-access-mnqmr\") pod \"d5a64752-7a83-4f0b-a1fa-39175a625238\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " Oct 30 05:37:57.964207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.964213 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-token\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-token\") pod \"d5a64752-7a83-4f0b-a1fa-39175a625238\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " Oct 30 05:37:57.964497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.964255 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a64752-7a83-4f0b-a1fa-39175a625238-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a64752-7a83-4f0b-a1fa-39175a625238-local-certs\") pod \"d5a64752-7a83-4f0b-a1fa-39175a625238\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " Oct 30 05:37:57.964497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.964293 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-cacerts\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-cacerts\") pod \"d5a64752-7a83-4f0b-a1fa-39175a625238\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " Oct 30 05:37:57.964497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.964320 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-kubeconfig\") pod \"d5a64752-7a83-4f0b-a1fa-39175a625238\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " Oct 30 05:37:57.964497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.964357 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-ca-configmap\") pod \"d5a64752-7a83-4f0b-a1fa-39175a625238\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " Oct 30 05:37:57.964497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.964382 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-dns-cert\") pod \"d5a64752-7a83-4f0b-a1fa-39175a625238\" (UID: \"d5a64752-7a83-4f0b-a1fa-39175a625238\") " Oct 30 05:37:57.965040 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.964996 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-ca-configmap" pod "d5a64752-7a83-4f0b-a1fa-39175a625238" (UID: "d5a64752-7a83-4f0b-a1fa-39175a625238"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:37:57.967421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.967396 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-token" pod "d5a64752-7a83-4f0b-a1fa-39175a625238" (UID: "d5a64752-7a83-4f0b-a1fa-39175a625238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:57.967421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.967402 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a64752-7a83-4f0b-a1fa-39175a625238-local-certs" pod "d5a64752-7a83-4f0b-a1fa-39175a625238" (UID: "d5a64752-7a83-4f0b-a1fa-39175a625238"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:57.967709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.967401 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-kubeconfig" pod "d5a64752-7a83-4f0b-a1fa-39175a625238" (UID: "d5a64752-7a83-4f0b-a1fa-39175a625238"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:57.967709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.967474 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-kube-api-access-mnqmr" pod "d5a64752-7a83-4f0b-a1fa-39175a625238" (UID: "d5a64752-7a83-4f0b-a1fa-39175a625238"). InnerVolumeSpecName "kube-api-access-mnqmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:57.967709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.967552 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-cacerts" pod "d5a64752-7a83-4f0b-a1fa-39175a625238" (UID: "d5a64752-7a83-4f0b-a1fa-39175a625238"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:57.967709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:57.967571 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-dns-cert" pod "d5a64752-7a83-4f0b-a1fa-39175a625238" (UID: "d5a64752-7a83-4f0b-a1fa-39175a625238"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:37:58.065178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.065101 5072 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-dns-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:37:58.065178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.065127 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnqmr\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-kube-api-access-mnqmr\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:37:58.065178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.065138 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:37:58.065178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.065147 5072 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a64752-7a83-4f0b-a1fa-39175a625238-local-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:37:58.065178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.065155 5072 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-cacerts\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:37:58.065178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.065162 5072 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-kubeconfig\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:37:58.065178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.065169 5072 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a64752-7a83-4f0b-a1fa-39175a625238-istio-csr-ca-configmap\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:37:58.544510 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.544481 5072 generic.go:356] "Generic (PLEG): container finished" podID="d5a64752-7a83-4f0b-a1fa-39175a625238" containerID="9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c" exitCode=0 Oct 30 05:37:58.544907 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.544535 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" Oct 30 05:37:58.544907 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.544555 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" event={"ID":"d5a64752-7a83-4f0b-a1fa-39175a625238","Type":"ContainerDied","Data":"9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c"} Oct 30 05:37:58.544907 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.544602 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s" event={"ID":"d5a64752-7a83-4f0b-a1fa-39175a625238","Type":"ContainerDied","Data":"86d0752b0d3abbb95992d306a949f9756173c2ad6b5fc538536f7c21490e10f3"} Oct 30 05:37:58.544907 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.544623 5072 scope.go:122] "RemoveContainer" containerID="9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c" Oct 30 05:37:58.553122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.553106 5072 scope.go:122] "RemoveContainer" containerID="9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c" Oct 30 05:37:58.553406 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:37:58.553386 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c\": container with ID starting with 9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c not found: ID does not exist" containerID="9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c" Oct 30 05:37:58.553451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.553416 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c"} err="failed to get container status \"9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c\": rpc error: code = NotFound desc = could not find container \"9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c\": container with ID starting with 9ad247630649f441cbb34452ef03679d681f3a2610d9c69e999e8ec76ed2fe4c not found: ID does not exist" Oct 30 05:37:58.591019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.590985 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s"] Oct 30 05:37:58.601407 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:37:58.601387 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wrg5s"] Oct 30 05:38:00.516941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:38:00.516907 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a64752-7a83-4f0b-a1fa-39175a625238" path="/var/lib/kubelet/pods/d5a64752-7a83-4f0b-a1fa-39175a625238/volumes" Oct 30 05:38:00.708412 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:38:00.708374 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:38:30.710095 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:38:30.710061 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:39:00.707993 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:00.707917 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:39:01.588125 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.588086 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l"] Oct 30 05:39:01.588498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.588484 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d5a64752-7a83-4f0b-a1fa-39175a625238" containerName="discovery" Oct 30 05:39:01.588552 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.588499 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a64752-7a83-4f0b-a1fa-39175a625238" containerName="discovery" Oct 30 05:39:01.588605 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.588596 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a64752-7a83-4f0b-a1fa-39175a625238" containerName="discovery" Oct 30 05:39:01.591735 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.591721 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" Oct 30 05:39:01.618117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.618078 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 05:39:01.618117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.618106 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 05:39:01.618117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.618118 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 05:39:01.618347 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.618185 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g6k4s\"" Oct 30 05:39:01.619911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.619884 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l"] Oct 30 05:39:01.730181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.730136 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7sw\" (UniqueName: \"kubernetes.io/projected/f1802508-d7a4-48d0-b5dc-89a480de16df-kube-api-access-hw7sw\") pod \"dns-operator-controller-manager-65bfd79598-fjf2l\" (UID: \"f1802508-d7a4-48d0-b5dc-89a480de16df\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" Oct 30 05:39:01.831098 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.831062 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7sw\" (UniqueName: \"kubernetes.io/projected/f1802508-d7a4-48d0-b5dc-89a480de16df-kube-api-access-hw7sw\") pod \"dns-operator-controller-manager-65bfd79598-fjf2l\" (UID: \"f1802508-d7a4-48d0-b5dc-89a480de16df\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" Oct 30 05:39:01.854566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.854542 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7sw\" (UniqueName: \"kubernetes.io/projected/f1802508-d7a4-48d0-b5dc-89a480de16df-kube-api-access-hw7sw\") pod \"dns-operator-controller-manager-65bfd79598-fjf2l\" (UID: \"f1802508-d7a4-48d0-b5dc-89a480de16df\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" Oct 30 05:39:01.904556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:01.904517 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" Oct 30 05:39:02.200193 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:02.200165 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l"] Oct 30 05:39:02.201980 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:39:02.201955 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1802508_d7a4_48d0_b5dc_89a480de16df.slice/crio-b8438c51b6b912224ad48fe3223c4104b748f937b9a092962a6e781a03c28bad WatchSource:0}: Error finding container b8438c51b6b912224ad48fe3223c4104b748f937b9a092962a6e781a03c28bad: Status 404 returned error can't find the container with id b8438c51b6b912224ad48fe3223c4104b748f937b9a092962a6e781a03c28bad Oct 30 05:39:02.819445 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:02.819411 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" event={"ID":"f1802508-d7a4-48d0-b5dc-89a480de16df","Type":"ContainerStarted","Data":"b8438c51b6b912224ad48fe3223c4104b748f937b9a092962a6e781a03c28bad"} Oct 30 05:39:03.695272 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:03.695242 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27"] Oct 30 05:39:03.704031 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:03.704002 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:03.733714 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:03.733682 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4fst6\"" Oct 30 05:39:03.735361 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:03.735336 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27"] Oct 30 05:39:03.850831 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:03.850794 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qq84\" (UniqueName: \"kubernetes.io/projected/23d3c922-45b1-483d-9be5-19c0da4794b9-kube-api-access-2qq84\") pod \"limitador-operator-controller-manager-856dd584c5-42l27\" (UID: \"23d3c922-45b1-483d-9be5-19c0da4794b9\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:03.952122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:03.952052 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qq84\" (UniqueName: \"kubernetes.io/projected/23d3c922-45b1-483d-9be5-19c0da4794b9-kube-api-access-2qq84\") pod \"limitador-operator-controller-manager-856dd584c5-42l27\" (UID: \"23d3c922-45b1-483d-9be5-19c0da4794b9\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:03.990288 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:03.990259 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qq84\" (UniqueName: \"kubernetes.io/projected/23d3c922-45b1-483d-9be5-19c0da4794b9-kube-api-access-2qq84\") pod \"limitador-operator-controller-manager-856dd584c5-42l27\" (UID: \"23d3c922-45b1-483d-9be5-19c0da4794b9\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:04.019079 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:04.019049 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:04.335404 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:04.335323 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27"] Oct 30 05:39:04.340571 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:39:04.340532 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d3c922_45b1_483d_9be5_19c0da4794b9.slice/crio-cb881606366cb63fe2a9ddc5fe66d0acb467c45fa5d547e51ddb7ae67bd2d81c WatchSource:0}: Error finding container cb881606366cb63fe2a9ddc5fe66d0acb467c45fa5d547e51ddb7ae67bd2d81c: Status 404 returned error can't find the container with id cb881606366cb63fe2a9ddc5fe66d0acb467c45fa5d547e51ddb7ae67bd2d81c Oct 30 05:39:04.831229 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:04.831182 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" event={"ID":"23d3c922-45b1-483d-9be5-19c0da4794b9","Type":"ContainerStarted","Data":"cb881606366cb63fe2a9ddc5fe66d0acb467c45fa5d547e51ddb7ae67bd2d81c"} Oct 30 05:39:04.833086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:04.833054 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" event={"ID":"f1802508-d7a4-48d0-b5dc-89a480de16df","Type":"ContainerStarted","Data":"40759f5464b9d0dcae6783b6c992c22b6ec6ce0f8dfd984702ad9b58ae22bad8"} Oct 30 05:39:04.833225 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:04.833195 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" Oct 30 05:39:04.896873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:04.896822 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" podStartSLOduration=1.554766228 podStartE2EDuration="3.896807173s" totalImagesPullingTime="2.342040945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:04.889282374 +0000 UTC m=+504.492971631" watchObservedRunningTime="2026-10-30 05:39:04.896807173 +0000 UTC m=+504.500496429" Oct 30 05:39:06.845104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:06.845071 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" event={"ID":"23d3c922-45b1-483d-9be5-19c0da4794b9","Type":"ContainerStarted","Data":"c0a7f2deabfa307337578b9f6d73a29eec3b62890f488b077b38501260d0971d"} Oct 30 05:39:06.845504 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:06.845197 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:06.899034 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:06.895747 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" podStartSLOduration=2.276114688 podStartE2EDuration="3.895728312s" totalImagesPullingTime="1.619613624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:06.893859447 +0000 UTC m=+506.497548739" watchObservedRunningTime="2026-10-30 05:39:06.895728312 +0000 UTC m=+506.499417571" Oct 30 05:39:08.689302 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.689266 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246"] Oct 30 05:39:08.692893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.692878 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:08.719300 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.719278 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2lbtb\"" Oct 30 05:39:08.720853 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.720835 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246"] Oct 30 05:39:08.797914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.797869 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4ccs\" (UniqueName: \"kubernetes.io/projected/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-kube-api-access-k4ccs\") pod \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" (UID: \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:08.797914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.797909 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" (UID: \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:08.899168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.899136 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ccs\" (UniqueName: \"kubernetes.io/projected/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-kube-api-access-k4ccs\") pod \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" (UID: \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:08.899168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.899172 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" (UID: \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:08.899524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.899508 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" (UID: \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:08.938027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:08.938001 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4ccs\" (UniqueName: \"kubernetes.io/projected/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-kube-api-access-k4ccs\") pod \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" (UID: \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:09.006116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:09.006047 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:09.338369 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:09.338282 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246"] Oct 30 05:39:09.340881 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:39:09.340852 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07b3d8f_b0d7_49c0_83e9_a8484d39228c.slice/crio-2c500b912612f0cf3c684801ab515eb9014857ef8fb961c9dc38506237b2a8a7 WatchSource:0}: Error finding container 2c500b912612f0cf3c684801ab515eb9014857ef8fb961c9dc38506237b2a8a7: Status 404 returned error can't find the container with id 2c500b912612f0cf3c684801ab515eb9014857ef8fb961c9dc38506237b2a8a7 Oct 30 05:39:09.860297 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:09.860247 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" event={"ID":"a07b3d8f-b0d7-49c0-83e9-a8484d39228c","Type":"ContainerStarted","Data":"2c500b912612f0cf3c684801ab515eb9014857ef8fb961c9dc38506237b2a8a7"} Oct 30 05:39:12.877556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:12.877507 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" event={"ID":"a07b3d8f-b0d7-49c0-83e9-a8484d39228c","Type":"ContainerStarted","Data":"6b0aa7fda3e00e831580e135d4bf738377956bad217f5219aaae56134149a859"} Oct 30 05:39:12.877932 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:12.877665 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:12.940483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:12.940411 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" podStartSLOduration=1.703395532 podStartE2EDuration="4.940396181s" totalImagesPullingTime="3.237000649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:12.938883828 +0000 UTC m=+512.542573087" watchObservedRunningTime="2026-10-30 05:39:12.940396181 +0000 UTC m=+512.544085437" Oct 30 05:39:15.841496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:15.841442 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" Oct 30 05:39:17.852281 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:17.852240 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:23.885188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:23.885156 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:27.675897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.675863 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246"] Oct 30 05:39:27.676567 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.676159 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" containerName="manager" containerID="cri-o://6b0aa7fda3e00e831580e135d4bf738377956bad217f5219aaae56134149a859" gracePeriod=2 Oct 30 05:39:27.681575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.681547 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246"] Oct 30 05:39:27.689160 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:27.689123 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:27.694300 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.694272 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8"] Oct 30 05:39:27.694700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.694687 5072 cpu_manager.go:414] "containerMap: removing container" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" containerName="manager" Oct 30 05:39:27.694753 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.694702 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" containerName="manager" Oct 30 05:39:27.694789 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.694781 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" containerName="manager" Oct 30 05:39:27.697881 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.697857 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:27.709900 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:27.709868 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:27.727567 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.727535 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8"] Oct 30 05:39:27.747451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.747422 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27"] Oct 30 05:39:27.747670 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.747648 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" containerName="manager" containerID="cri-o://c0a7f2deabfa307337578b9f6d73a29eec3b62890f488b077b38501260d0971d" gracePeriod=2 Oct 30 05:39:27.758948 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.758919 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27"] Oct 30 05:39:27.770867 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:27.770830 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:27.775279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.775251 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wqsx8\" (UID: \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:27.775398 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.775295 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvnc\" (UniqueName: \"kubernetes.io/projected/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-kube-api-access-qcvnc\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wqsx8\" (UID: \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:27.776986 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.776962 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j"] Oct 30 05:39:27.777416 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.777399 5072 cpu_manager.go:414] "containerMap: removing container" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" containerName="manager" Oct 30 05:39:27.777530 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.777419 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" containerName="manager" Oct 30 05:39:27.777591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.777564 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" containerName="manager" Oct 30 05:39:27.780917 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.780898 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" Oct 30 05:39:27.782652 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:27.782619 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-42l27\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:27.806194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.806161 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j"] Oct 30 05:39:27.841883 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:27.841851 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:27.854564 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:27.854425 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-42l27\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:27.875999 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.875966 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqp8\" (UniqueName: \"kubernetes.io/projected/3d979a36-0e1f-45eb-8fef-34fdecd4ba8d-kube-api-access-gjqp8\") pod \"limitador-operator-controller-manager-856dd584c5-qh68j\" (UID: \"3d979a36-0e1f-45eb-8fef-34fdecd4ba8d\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" Oct 30 05:39:27.876133 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.876113 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wqsx8\" (UID: \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:27.876194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.876152 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvnc\" (UniqueName: \"kubernetes.io/projected/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-kube-api-access-qcvnc\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wqsx8\" (UID: \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:27.876522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.876504 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wqsx8\" (UID: \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:27.903032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.902998 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvnc\" (UniqueName: \"kubernetes.io/projected/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-kube-api-access-qcvnc\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wqsx8\" (UID: \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:27.950794 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.950765 5072 generic.go:356] "Generic (PLEG): container finished" podID="23d3c922-45b1-483d-9be5-19c0da4794b9" containerID="c0a7f2deabfa307337578b9f6d73a29eec3b62890f488b077b38501260d0971d" exitCode=0 Oct 30 05:39:27.952808 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.952777 5072 generic.go:356] "Generic (PLEG): container finished" podID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" containerID="6b0aa7fda3e00e831580e135d4bf738377956bad217f5219aaae56134149a859" exitCode=0 Oct 30 05:39:27.976814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.976773 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqp8\" (UniqueName: \"kubernetes.io/projected/3d979a36-0e1f-45eb-8fef-34fdecd4ba8d-kube-api-access-gjqp8\") pod \"limitador-operator-controller-manager-856dd584c5-qh68j\" (UID: \"3d979a36-0e1f-45eb-8fef-34fdecd4ba8d\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" Oct 30 05:39:27.997868 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:27.997847 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:28.000409 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.000391 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:28.005337 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.005320 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqp8\" (UniqueName: \"kubernetes.io/projected/3d979a36-0e1f-45eb-8fef-34fdecd4ba8d-kube-api-access-gjqp8\") pod \"limitador-operator-controller-manager-856dd584c5-qh68j\" (UID: \"3d979a36-0e1f-45eb-8fef-34fdecd4ba8d\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" Oct 30 05:39:28.009956 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:28.009932 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:28.029884 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:28.029854 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-42l27\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:28.041776 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:28.041749 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-p4246\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:28.053907 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.053889 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:28.053994 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:28.053942 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-42l27\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:28.078187 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.078162 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d3c922-45b1-483d-9be5-19c0da4794b9-kube-api-access-2qq84\" (UniqueName: \"kubernetes.io/projected/23d3c922-45b1-483d-9be5-19c0da4794b9-kube-api-access-2qq84\") pod \"23d3c922-45b1-483d-9be5-19c0da4794b9\" (UID: \"23d3c922-45b1-483d-9be5-19c0da4794b9\") " Oct 30 05:39:28.078289 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.078196 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-kube-api-access-k4ccs\" (UniqueName: \"kubernetes.io/projected/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-kube-api-access-k4ccs\") pod \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\" (UID: \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\") " Oct 30 05:39:28.078289 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.078265 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-extensions-socket-volume\") pod \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\" (UID: \"a07b3d8f-b0d7-49c0-83e9-a8484d39228c\") " Oct 30 05:39:28.078782 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.078758 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-extensions-socket-volume" pod "a07b3d8f-b0d7-49c0-83e9-a8484d39228c" (UID: "a07b3d8f-b0d7-49c0-83e9-a8484d39228c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:28.080555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.080522 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-kube-api-access-k4ccs" pod "a07b3d8f-b0d7-49c0-83e9-a8484d39228c" (UID: "a07b3d8f-b0d7-49c0-83e9-a8484d39228c"). InnerVolumeSpecName "kube-api-access-k4ccs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:28.080641 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.080594 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d3c922-45b1-483d-9be5-19c0da4794b9-kube-api-access-2qq84" pod "23d3c922-45b1-483d-9be5-19c0da4794b9" (UID: "23d3c922-45b1-483d-9be5-19c0da4794b9"). InnerVolumeSpecName "kube-api-access-2qq84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:28.134436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.134407 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" Oct 30 05:39:28.179295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.179223 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qq84\" (UniqueName: \"kubernetes.io/projected/23d3c922-45b1-483d-9be5-19c0da4794b9-kube-api-access-2qq84\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:39:28.179295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.179260 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4ccs\" (UniqueName: \"kubernetes.io/projected/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-kube-api-access-k4ccs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:39:28.179295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.179275 5072 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a07b3d8f-b0d7-49c0-83e9-a8484d39228c-extensions-socket-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:39:28.378287 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.378262 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8"] Oct 30 05:39:28.380387 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:39:28.380359 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6e7364_eefb_42e9_a20f_61dd65ec8b90.slice/crio-ddb7ba50c3b2e7458023503a22e3990e368bc4c15285a0f31ee5044e98e76456 WatchSource:0}: Error finding container ddb7ba50c3b2e7458023503a22e3990e368bc4c15285a0f31ee5044e98e76456: Status 404 returned error can't find the container with id ddb7ba50c3b2e7458023503a22e3990e368bc4c15285a0f31ee5044e98e76456 Oct 30 05:39:28.443396 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.443367 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j"] Oct 30 05:39:28.446767 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:39:28.446743 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d979a36_0e1f_45eb_8fef_34fdecd4ba8d.slice/crio-c34d3e65411d6cb43b7170706dad3de023120eefb1a24f710d77bae3a93b81d6 WatchSource:0}: Error finding container c34d3e65411d6cb43b7170706dad3de023120eefb1a24f710d77bae3a93b81d6: Status 404 returned error can't find the container with id c34d3e65411d6cb43b7170706dad3de023120eefb1a24f710d77bae3a93b81d6 Oct 30 05:39:28.517997 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.517970 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" path="/var/lib/kubelet/pods/23d3c922-45b1-483d-9be5-19c0da4794b9/volumes" Oct 30 05:39:28.518419 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.518391 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07b3d8f-b0d7-49c0-83e9-a8484d39228c" path="/var/lib/kubelet/pods/a07b3d8f-b0d7-49c0-83e9-a8484d39228c/volumes" Oct 30 05:39:28.850380 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.850338 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" podUID="23d3c922-45b1-483d-9be5-19c0da4794b9" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:39:28.959281 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.959194 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-42l27" Oct 30 05:39:28.959281 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.959209 5072 scope.go:122] "RemoveContainer" containerID="c0a7f2deabfa307337578b9f6d73a29eec3b62890f488b077b38501260d0971d" Oct 30 05:39:28.962328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.962299 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" event={"ID":"3d979a36-0e1f-45eb-8fef-34fdecd4ba8d","Type":"ContainerStarted","Data":"4557ca37ef06b22d37c072b8cb471c59c29ec54a77011d64f439091094c1a8fd"} Oct 30 05:39:28.962483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.962351 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" event={"ID":"3d979a36-0e1f-45eb-8fef-34fdecd4ba8d","Type":"ContainerStarted","Data":"c34d3e65411d6cb43b7170706dad3de023120eefb1a24f710d77bae3a93b81d6"} Oct 30 05:39:28.962745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.962729 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" Oct 30 05:39:28.964219 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.964196 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" event={"ID":"6c6e7364-eefb-42e9-a20f-61dd65ec8b90","Type":"ContainerStarted","Data":"9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69"} Oct 30 05:39:28.964327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.964248 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:28.964327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.964282 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" event={"ID":"6c6e7364-eefb-42e9-a20f-61dd65ec8b90","Type":"ContainerStarted","Data":"ddb7ba50c3b2e7458023503a22e3990e368bc4c15285a0f31ee5044e98e76456"} Oct 30 05:39:28.965637 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.965619 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-p4246" Oct 30 05:39:28.968612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:28.968597 5072 scope.go:122] "RemoveContainer" containerID="6b0aa7fda3e00e831580e135d4bf738377956bad217f5219aaae56134149a859" Oct 30 05:39:29.017315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:29.017263 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" podStartSLOduration=2.01724859 podStartE2EDuration="2.01724859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:29.015490051 +0000 UTC m=+528.619179305" watchObservedRunningTime="2026-10-30 05:39:29.01724859 +0000 UTC m=+528.620937846" Oct 30 05:39:29.067910 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:29.067804 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" podStartSLOduration=2.067788281 podStartE2EDuration="2.067788281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:29.066905971 +0000 UTC m=+528.670595228" watchObservedRunningTime="2026-10-30 05:39:29.067788281 +0000 UTC m=+528.671477539" Oct 30 05:39:30.707964 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:30.707921 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:39:39.975128 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:39.975101 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:39.975619 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:39.975262 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-qh68j" Oct 30 05:39:44.348308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.348273 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8"] Oct 30 05:39:44.348779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.348529 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" podUID="6c6e7364-eefb-42e9-a20f-61dd65ec8b90" containerName="manager" containerID="cri-o://9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69" gracePeriod=10 Oct 30 05:39:44.614217 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.614185 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:44.645994 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.645962 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv"] Oct 30 05:39:44.646337 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.646326 5072 cpu_manager.go:414] "containerMap: removing container" podUID="6c6e7364-eefb-42e9-a20f-61dd65ec8b90" containerName="manager" Oct 30 05:39:44.646376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.646338 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6e7364-eefb-42e9-a20f-61dd65ec8b90" containerName="manager" Oct 30 05:39:44.646422 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.646413 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6e7364-eefb-42e9-a20f-61dd65ec8b90" containerName="manager" Oct 30 05:39:44.648251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.648237 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:39:44.676065 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.676034 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv"] Oct 30 05:39:44.723042 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.723008 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-extensions-socket-volume\") pod \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\" (UID: \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\") " Oct 30 05:39:44.723222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.723069 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-kube-api-access-qcvnc\" (UniqueName: \"kubernetes.io/projected/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-kube-api-access-qcvnc\") pod \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\" (UID: \"6c6e7364-eefb-42e9-a20f-61dd65ec8b90\") " Oct 30 05:39:44.723391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.723324 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/337e2886-d4e4-4944-abcf-dcea923a1e7b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4c4mv\" (UID: \"337e2886-d4e4-4944-abcf-dcea923a1e7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:39:44.723391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.723365 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87mvw\" (UniqueName: \"kubernetes.io/projected/337e2886-d4e4-4944-abcf-dcea923a1e7b-kube-api-access-87mvw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4c4mv\" (UID: \"337e2886-d4e4-4944-abcf-dcea923a1e7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:39:44.723391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.723372 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-extensions-socket-volume" pod "6c6e7364-eefb-42e9-a20f-61dd65ec8b90" (UID: "6c6e7364-eefb-42e9-a20f-61dd65ec8b90"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:44.723614 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.723534 5072 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-extensions-socket-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:39:44.725331 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.725305 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-kube-api-access-qcvnc" pod "6c6e7364-eefb-42e9-a20f-61dd65ec8b90" (UID: "6c6e7364-eefb-42e9-a20f-61dd65ec8b90"). InnerVolumeSpecName "kube-api-access-qcvnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:44.824305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.824269 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/337e2886-d4e4-4944-abcf-dcea923a1e7b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4c4mv\" (UID: \"337e2886-d4e4-4944-abcf-dcea923a1e7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:39:44.824305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.824312 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mvw\" (UniqueName: \"kubernetes.io/projected/337e2886-d4e4-4944-abcf-dcea923a1e7b-kube-api-access-87mvw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4c4mv\" (UID: \"337e2886-d4e4-4944-abcf-dcea923a1e7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:39:44.824523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.824401 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcvnc\" (UniqueName: \"kubernetes.io/projected/6c6e7364-eefb-42e9-a20f-61dd65ec8b90-kube-api-access-qcvnc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:39:44.824679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.824659 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/337e2886-d4e4-4944-abcf-dcea923a1e7b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4c4mv\" (UID: \"337e2886-d4e4-4944-abcf-dcea923a1e7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:39:44.850292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.850230 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87mvw\" (UniqueName: \"kubernetes.io/projected/337e2886-d4e4-4944-abcf-dcea923a1e7b-kube-api-access-87mvw\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4c4mv\" (UID: \"337e2886-d4e4-4944-abcf-dcea923a1e7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:39:44.960469 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:44.960435 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:39:45.047851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.047814 5072 generic.go:356] "Generic (PLEG): container finished" podID="6c6e7364-eefb-42e9-a20f-61dd65ec8b90" containerID="9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69" exitCode=0 Oct 30 05:39:45.048012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.047863 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" Oct 30 05:39:45.048012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.047958 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" event={"ID":"6c6e7364-eefb-42e9-a20f-61dd65ec8b90","Type":"ContainerDied","Data":"9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69"} Oct 30 05:39:45.048012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.048010 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8" event={"ID":"6c6e7364-eefb-42e9-a20f-61dd65ec8b90","Type":"ContainerDied","Data":"ddb7ba50c3b2e7458023503a22e3990e368bc4c15285a0f31ee5044e98e76456"} Oct 30 05:39:45.048199 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.048041 5072 scope.go:122] "RemoveContainer" containerID="9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69" Oct 30 05:39:45.058350 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.058331 5072 scope.go:122] "RemoveContainer" containerID="9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69" Oct 30 05:39:45.058677 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:39:45.058656 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69\": container with ID starting with 9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69 not found: ID does not exist" containerID="9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69" Oct 30 05:39:45.058754 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.058687 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69"} err="failed to get container status \"9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69\": rpc error: code = NotFound desc = could not find container \"9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69\": container with ID starting with 9ac9b068cca95598dd54092eb611c0e62642bfd65b3272b66b4e9e27fff09b69 not found: ID does not exist" Oct 30 05:39:45.127191 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.127153 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8"] Oct 30 05:39:45.131535 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.131511 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wqsx8"] Oct 30 05:39:45.275231 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:45.275194 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv"] Oct 30 05:39:45.277652 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:39:45.277627 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337e2886_d4e4_4944_abcf_dcea923a1e7b.slice/crio-4d223c9b1d30ef70f3b21063c80c7914cd8cb0bc3e11f7dcfd826bc192d9ce6b WatchSource:0}: Error finding container 4d223c9b1d30ef70f3b21063c80c7914cd8cb0bc3e11f7dcfd826bc192d9ce6b: Status 404 returned error can't find the container with id 4d223c9b1d30ef70f3b21063c80c7914cd8cb0bc3e11f7dcfd826bc192d9ce6b Oct 30 05:39:46.053783 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:46.053747 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" event={"ID":"337e2886-d4e4-4944-abcf-dcea923a1e7b","Type":"ContainerStarted","Data":"b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55"} Oct 30 05:39:46.054202 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:46.053805 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" event={"ID":"337e2886-d4e4-4944-abcf-dcea923a1e7b","Type":"ContainerStarted","Data":"4d223c9b1d30ef70f3b21063c80c7914cd8cb0bc3e11f7dcfd826bc192d9ce6b"} Oct 30 05:39:46.054202 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:46.053850 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:39:46.118976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:46.118919 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" podStartSLOduration=2.118904985 podStartE2EDuration="2.118904985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:46.107114379 +0000 UTC m=+545.710803655" watchObservedRunningTime="2026-10-30 05:39:46.118904985 +0000 UTC m=+545.722594238" Oct 30 05:39:46.515828 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:46.515794 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6e7364-eefb-42e9-a20f-61dd65ec8b90" path="/var/lib/kubelet/pods/6c6e7364-eefb-42e9-a20f-61dd65ec8b90/volumes" Oct 30 05:39:57.061920 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:39:57.061889 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 05:40:00.709384 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:40:00.709278 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:40:17.076379 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.076341 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-28j78"] Oct 30 05:40:17.080201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.080181 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:17.100661 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.100636 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 05:40:17.107095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.107070 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d973680-c484-4676-84e4-3241c139ac6f-config-file\") pod \"limitador-limitador-74dfcfb894-28j78\" (UID: \"5d973680-c484-4676-84e4-3241c139ac6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:17.107196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.107124 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2nvh\" (UniqueName: \"kubernetes.io/projected/5d973680-c484-4676-84e4-3241c139ac6f-kube-api-access-r2nvh\") pod \"limitador-limitador-74dfcfb894-28j78\" (UID: \"5d973680-c484-4676-84e4-3241c139ac6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:17.107596 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.107579 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pljfh\"" Oct 30 05:40:17.109818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.109794 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-28j78"] Oct 30 05:40:17.207720 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.207688 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d973680-c484-4676-84e4-3241c139ac6f-config-file\") pod \"limitador-limitador-74dfcfb894-28j78\" (UID: \"5d973680-c484-4676-84e4-3241c139ac6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:17.207889 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.207727 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2nvh\" (UniqueName: \"kubernetes.io/projected/5d973680-c484-4676-84e4-3241c139ac6f-kube-api-access-r2nvh\") pod \"limitador-limitador-74dfcfb894-28j78\" (UID: \"5d973680-c484-4676-84e4-3241c139ac6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:17.208311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.208293 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d973680-c484-4676-84e4-3241c139ac6f-config-file\") pod \"limitador-limitador-74dfcfb894-28j78\" (UID: \"5d973680-c484-4676-84e4-3241c139ac6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:17.212975 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.212950 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-28j78"] Oct 30 05:40:17.230333 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.230303 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2nvh\" (UniqueName: \"kubernetes.io/projected/5d973680-c484-4676-84e4-3241c139ac6f-kube-api-access-r2nvh\") pod \"limitador-limitador-74dfcfb894-28j78\" (UID: \"5d973680-c484-4676-84e4-3241c139ac6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:17.393752 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.393714 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:17.689774 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:17.689751 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-28j78"] Oct 30 05:40:17.692392 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:40:17.692365 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d973680_c484_4676_84e4_3241c139ac6f.slice/crio-68ee09a9746a831535df41273b4b8647814b976282c00701810a969826219bf1 WatchSource:0}: Error finding container 68ee09a9746a831535df41273b4b8647814b976282c00701810a969826219bf1: Status 404 returned error can't find the container with id 68ee09a9746a831535df41273b4b8647814b976282c00701810a969826219bf1 Oct 30 05:40:18.200186 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:18.200156 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" event={"ID":"5d973680-c484-4676-84e4-3241c139ac6f","Type":"ContainerStarted","Data":"68ee09a9746a831535df41273b4b8647814b976282c00701810a969826219bf1"} Oct 30 05:40:19.206439 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:19.206402 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" event={"ID":"5d973680-c484-4676-84e4-3241c139ac6f","Type":"ContainerStarted","Data":"0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a"} Oct 30 05:40:19.206941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:19.206484 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:19.246980 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:19.246931 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" podStartSLOduration=1.2895527 podStartE2EDuration="2.246917594s" totalImagesPullingTime="957.364894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:19.245495347 +0000 UTC m=+578.849184610" watchObservedRunningTime="2026-10-30 05:40:19.246917594 +0000 UTC m=+578.850606877" Oct 30 05:40:30.212010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:30.211981 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:30.640780 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:30.640748 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-28j78"] Oct 30 05:40:30.640982 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:30.640957 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" podUID="5d973680-c484-4676-84e4-3241c139ac6f" containerName="limitador" containerID="cri-o://0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a" gracePeriod=30 Oct 30 05:40:30.708895 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:40:30.708857 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:40:31.187129 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.187099 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:31.230572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.230501 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d973680-c484-4676-84e4-3241c139ac6f-kube-api-access-r2nvh\" (UniqueName: \"kubernetes.io/projected/5d973680-c484-4676-84e4-3241c139ac6f-kube-api-access-r2nvh\") pod \"5d973680-c484-4676-84e4-3241c139ac6f\" (UID: \"5d973680-c484-4676-84e4-3241c139ac6f\") " Oct 30 05:40:31.230572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.230543 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d973680-c484-4676-84e4-3241c139ac6f-config-file\" (UniqueName: \"kubernetes.io/configmap/5d973680-c484-4676-84e4-3241c139ac6f-config-file\") pod \"5d973680-c484-4676-84e4-3241c139ac6f\" (UID: \"5d973680-c484-4676-84e4-3241c139ac6f\") " Oct 30 05:40:31.231052 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.230921 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d973680-c484-4676-84e4-3241c139ac6f-config-file" pod "5d973680-c484-4676-84e4-3241c139ac6f" (UID: "5d973680-c484-4676-84e4-3241c139ac6f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:40:31.233395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.233365 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d973680-c484-4676-84e4-3241c139ac6f-kube-api-access-r2nvh" pod "5d973680-c484-4676-84e4-3241c139ac6f" (UID: "5d973680-c484-4676-84e4-3241c139ac6f"). InnerVolumeSpecName "kube-api-access-r2nvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:31.265944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.265918 5072 generic.go:356] "Generic (PLEG): container finished" podID="5d973680-c484-4676-84e4-3241c139ac6f" containerID="0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a" exitCode=0 Oct 30 05:40:31.266066 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.265965 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" Oct 30 05:40:31.266066 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.265966 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" event={"ID":"5d973680-c484-4676-84e4-3241c139ac6f","Type":"ContainerDied","Data":"0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a"} Oct 30 05:40:31.266182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.266107 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-28j78" event={"ID":"5d973680-c484-4676-84e4-3241c139ac6f","Type":"ContainerDied","Data":"68ee09a9746a831535df41273b4b8647814b976282c00701810a969826219bf1"} Oct 30 05:40:31.266182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.266146 5072 scope.go:122] "RemoveContainer" containerID="0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a" Oct 30 05:40:31.275276 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.275253 5072 scope.go:122] "RemoveContainer" containerID="0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a" Oct 30 05:40:31.275595 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:40:31.275574 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a\": container with ID starting with 0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a not found: ID does not exist" containerID="0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a" Oct 30 05:40:31.275666 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.275605 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a"} err="failed to get container status \"0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a\": rpc error: code = NotFound desc = could not find container \"0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a\": container with ID starting with 0ffbcae424a145066a9711753f9e59d03c6e6f67fdff32d7f53ba730af68719a not found: ID does not exist" Oct 30 05:40:31.312126 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.312099 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-28j78"] Oct 30 05:40:31.316571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.316547 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-28j78"] Oct 30 05:40:31.331116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.331094 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2nvh\" (UniqueName: \"kubernetes.io/projected/5d973680-c484-4676-84e4-3241c139ac6f-kube-api-access-r2nvh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:40:31.331116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:31.331115 5072 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5d973680-c484-4676-84e4-3241c139ac6f-config-file\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:40:32.515802 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:32.515771 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d973680-c484-4676-84e4-3241c139ac6f" path="/var/lib/kubelet/pods/5d973680-c484-4676-84e4-3241c139ac6f/volumes" Oct 30 05:40:40.564751 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:40.564722 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:40:40.565370 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:40.565346 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:40:59.278016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.277976 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-vjd85"] Oct 30 05:40:59.285025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.284997 5072 cpu_manager.go:414] "containerMap: removing container" podUID="5d973680-c484-4676-84e4-3241c139ac6f" containerName="limitador" Oct 30 05:40:59.285025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.285023 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d973680-c484-4676-84e4-3241c139ac6f" containerName="limitador" Oct 30 05:40:59.285214 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.285103 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d973680-c484-4676-84e4-3241c139ac6f" containerName="limitador" Oct 30 05:40:59.288732 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.288712 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.303747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.303725 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 05:40:59.304022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.304006 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6ttmw\"" Oct 30 05:40:59.313345 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.313325 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 05:40:59.313911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.313875 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 05:40:59.315417 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.315399 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-vjd85"] Oct 30 05:40:59.477512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.477444 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-metrics-certs\") pod \"maas-controller-577f596567-vjd85\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.477512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.477502 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllrv\" (UniqueName: \"kubernetes.io/projected/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-kube-api-access-bllrv\") pod \"maas-controller-577f596567-vjd85\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.477767 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.477526 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-webhook-cert\") pod \"maas-controller-577f596567-vjd85\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.579015 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.578907 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-metrics-certs\") pod \"maas-controller-577f596567-vjd85\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.579015 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.578957 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllrv\" (UniqueName: \"kubernetes.io/projected/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-kube-api-access-bllrv\") pod \"maas-controller-577f596567-vjd85\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.579015 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.578986 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-webhook-cert\") pod \"maas-controller-577f596567-vjd85\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.581524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.581501 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-webhook-cert\") pod \"maas-controller-577f596567-vjd85\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.581642 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.581621 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-metrics-certs\") pod \"maas-controller-577f596567-vjd85\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.600938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.600908 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllrv\" (UniqueName: \"kubernetes.io/projected/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-kube-api-access-bllrv\") pod \"maas-controller-577f596567-vjd85\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.602579 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.602564 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:40:59.655675 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.655649 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b97b5857-2ppc5"] Oct 30 05:40:59.659883 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.659848 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.680983 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.680951 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b97b5857-2ppc5"] Oct 30 05:40:59.782265 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.782232 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-webhook-cert\") pod \"maas-controller-7b97b5857-2ppc5\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.782265 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.782265 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-metrics-certs\") pod \"maas-controller-7b97b5857-2ppc5\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.782531 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.782404 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvthp\" (UniqueName: \"kubernetes.io/projected/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-kube-api-access-cvthp\") pod \"maas-controller-7b97b5857-2ppc5\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.883585 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.883547 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvthp\" (UniqueName: \"kubernetes.io/projected/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-kube-api-access-cvthp\") pod \"maas-controller-7b97b5857-2ppc5\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.883772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.883604 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-webhook-cert\") pod \"maas-controller-7b97b5857-2ppc5\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.883772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.883624 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-metrics-certs\") pod \"maas-controller-7b97b5857-2ppc5\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.886241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.886218 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-metrics-certs\") pod \"maas-controller-7b97b5857-2ppc5\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.886241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.886233 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-webhook-cert\") pod \"maas-controller-7b97b5857-2ppc5\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.903313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.903296 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-vjd85"] Oct 30 05:40:59.905574 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:40:59.905537 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35205cc5_e7a0_4d09_ab49_ba595b9e1fdb.slice/crio-df0e0fb42fa2a31258de4bc308891e7f13571adca20e886b5fde170cb64763b1 WatchSource:0}: Error finding container df0e0fb42fa2a31258de4bc308891e7f13571adca20e886b5fde170cb64763b1: Status 404 returned error can't find the container with id df0e0fb42fa2a31258de4bc308891e7f13571adca20e886b5fde170cb64763b1 Oct 30 05:40:59.908286 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.908247 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvthp\" (UniqueName: \"kubernetes.io/projected/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-kube-api-access-cvthp\") pod \"maas-controller-7b97b5857-2ppc5\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:40:59.979165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:40:59.979139 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:41:00.277321 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:00.277294 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b97b5857-2ppc5"] Oct 30 05:41:00.279680 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:41:00.279650 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a793bc_54e2_4bb3_aaac_ec7a7d94467c.slice/crio-4114062471c03b9e08859adb30fc6e812fa29987bb5d4dec31f8cf3828b318e4 WatchSource:0}: Error finding container 4114062471c03b9e08859adb30fc6e812fa29987bb5d4dec31f8cf3828b318e4: Status 404 returned error can't find the container with id 4114062471c03b9e08859adb30fc6e812fa29987bb5d4dec31f8cf3828b318e4 Oct 30 05:41:00.398656 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:00.398617 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-2ppc5" event={"ID":"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c","Type":"ContainerStarted","Data":"4114062471c03b9e08859adb30fc6e812fa29987bb5d4dec31f8cf3828b318e4"} Oct 30 05:41:00.400368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:00.400339 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-vjd85" event={"ID":"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb","Type":"ContainerStarted","Data":"df0e0fb42fa2a31258de4bc308891e7f13571adca20e886b5fde170cb64763b1"} Oct 30 05:41:00.724298 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:41:00.724255 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:41:03.420553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:03.420511 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-2ppc5" event={"ID":"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c","Type":"ContainerStarted","Data":"a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c"} Oct 30 05:41:03.421021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:03.420612 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:41:03.422520 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:03.422491 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-vjd85" event={"ID":"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb","Type":"ContainerStarted","Data":"e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64"} Oct 30 05:41:03.422667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:03.422613 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:41:03.468643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:03.468597 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b97b5857-2ppc5" podStartSLOduration=1.89991998 podStartE2EDuration="4.468583642s" totalImagesPullingTime="2.568663662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:03.464575563 +0000 UTC m=+623.068264819" watchObservedRunningTime="2026-10-30 05:41:03.468583642 +0000 UTC m=+623.072272899" Oct 30 05:41:03.510162 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:03.510108 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-vjd85" podStartSLOduration=1.935520103 podStartE2EDuration="4.510092187s" totalImagesPullingTime="2.574572084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:03.509331421 +0000 UTC m=+623.113020679" watchObservedRunningTime="2026-10-30 05:41:03.510092187 +0000 UTC m=+623.113781444" Oct 30 05:41:06.194657 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.194618 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt"] Oct 30 05:41:06.199043 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.199025 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.224965 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.224946 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 05:41:06.232227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.232207 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-tbfsl\"" Oct 30 05:41:06.239789 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.239755 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt"] Oct 30 05:41:06.241194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.241152 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 05:41:06.241288 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.241233 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 05:41:06.244741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.244715 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f59ca931-b8ac-4755-b89e-d06668fc00e2-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.244837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.244752 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f59ca931-b8ac-4755-b89e-d06668fc00e2-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.244837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.244814 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxspb\" (UniqueName: \"kubernetes.io/projected/f59ca931-b8ac-4755-b89e-d06668fc00e2-kube-api-access-vxspb\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.244939 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.244918 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f59ca931-b8ac-4755-b89e-d06668fc00e2-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.345521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.345474 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f59ca931-b8ac-4755-b89e-d06668fc00e2-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.345521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.345526 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f59ca931-b8ac-4755-b89e-d06668fc00e2-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.345792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.345572 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxspb\" (UniqueName: \"kubernetes.io/projected/f59ca931-b8ac-4755-b89e-d06668fc00e2-kube-api-access-vxspb\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.345792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.345611 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f59ca931-b8ac-4755-b89e-d06668fc00e2-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.346352 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.346323 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f59ca931-b8ac-4755-b89e-d06668fc00e2-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.346705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.346646 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f59ca931-b8ac-4755-b89e-d06668fc00e2-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.348234 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.348215 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f59ca931-b8ac-4755-b89e-d06668fc00e2-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.382568 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.382526 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxspb\" (UniqueName: \"kubernetes.io/projected/f59ca931-b8ac-4755-b89e-d06668fc00e2-kube-api-access-vxspb\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b6gpt\" (UID: \"f59ca931-b8ac-4755-b89e-d06668fc00e2\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.513900 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.513818 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:06.813574 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:06.813483 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt"] Oct 30 05:41:06.816920 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:41:06.816895 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59ca931_b8ac_4755_b89e_d06668fc00e2.slice/crio-7752e283fd47d6c23c6335c9ee918820466cf99da7a6f9f4a56330468e5ef232 WatchSource:0}: Error finding container 7752e283fd47d6c23c6335c9ee918820466cf99da7a6f9f4a56330468e5ef232: Status 404 returned error can't find the container with id 7752e283fd47d6c23c6335c9ee918820466cf99da7a6f9f4a56330468e5ef232 Oct 30 05:41:07.443622 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:07.443589 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" event={"ID":"f59ca931-b8ac-4755-b89e-d06668fc00e2","Type":"ContainerStarted","Data":"7752e283fd47d6c23c6335c9ee918820466cf99da7a6f9f4a56330468e5ef232"} Oct 30 05:41:14.431426 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.431394 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:41:14.431845 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.431594 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 05:41:14.507386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.507352 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-vjd85"] Oct 30 05:41:14.507628 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.507581 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-vjd85" podUID="35205cc5-e7a0-4d09-ab49-ba595b9e1fdb" containerName="manager" containerID="cri-o://e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64" gracePeriod=30 Oct 30 05:41:14.784341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.784313 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:41:14.835503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.835432 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-kube-api-access-bllrv\" (UniqueName: \"kubernetes.io/projected/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-kube-api-access-bllrv\") pod \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " Oct 30 05:41:14.835698 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.835521 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-metrics-certs\") pod \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " Oct 30 05:41:14.835698 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.835568 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-webhook-cert\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-webhook-cert\") pod \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\" (UID: \"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb\") " Oct 30 05:41:14.838283 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.838247 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-kube-api-access-bllrv" pod "35205cc5-e7a0-4d09-ab49-ba595b9e1fdb" (UID: "35205cc5-e7a0-4d09-ab49-ba595b9e1fdb"). InnerVolumeSpecName "kube-api-access-bllrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:41:14.838283 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.838277 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-webhook-cert" pod "35205cc5-e7a0-4d09-ab49-ba595b9e1fdb" (UID: "35205cc5-e7a0-4d09-ab49-ba595b9e1fdb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:41:14.838687 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.838645 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-metrics-certs" pod "35205cc5-e7a0-4d09-ab49-ba595b9e1fdb" (UID: "35205cc5-e7a0-4d09-ab49-ba595b9e1fdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:41:14.937090 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.937056 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bllrv\" (UniqueName: \"kubernetes.io/projected/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-kube-api-access-bllrv\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:41:14.937090 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.937085 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:41:14.937090 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:14.937096 5072 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb-webhook-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:41:15.493415 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:15.493379 5072 generic.go:356] "Generic (PLEG): container finished" podID="35205cc5-e7a0-4d09-ab49-ba595b9e1fdb" containerID="e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64" exitCode=0 Oct 30 05:41:15.493891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:15.493429 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-vjd85" Oct 30 05:41:15.493891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:15.493486 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-vjd85" event={"ID":"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb","Type":"ContainerDied","Data":"e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64"} Oct 30 05:41:15.493891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:15.493557 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-vjd85" event={"ID":"35205cc5-e7a0-4d09-ab49-ba595b9e1fdb","Type":"ContainerDied","Data":"df0e0fb42fa2a31258de4bc308891e7f13571adca20e886b5fde170cb64763b1"} Oct 30 05:41:15.493891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:15.493584 5072 scope.go:122] "RemoveContainer" containerID="e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64" Oct 30 05:41:15.503841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:15.503823 5072 scope.go:122] "RemoveContainer" containerID="e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64" Oct 30 05:41:15.504163 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:41:15.504135 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64\": container with ID starting with e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64 not found: ID does not exist" containerID="e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64" Oct 30 05:41:15.504290 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:15.504179 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64"} err="failed to get container status \"e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64\": rpc error: code = NotFound desc = could not find container \"e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64\": container with ID starting with e3dc6f7dade34ba544b4f22ac1d38d9848151be508d8aa8a4901cd50fd5dae64 not found: ID does not exist" Oct 30 05:41:15.547079 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:15.542704 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-vjd85"] Oct 30 05:41:15.552280 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:15.552233 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-vjd85"] Oct 30 05:41:16.516642 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:16.516611 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35205cc5-e7a0-4d09-ab49-ba595b9e1fdb" path="/var/lib/kubelet/pods/35205cc5-e7a0-4d09-ab49-ba595b9e1fdb/volumes" Oct 30 05:41:17.506395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:17.506361 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" event={"ID":"f59ca931-b8ac-4755-b89e-d06668fc00e2","Type":"ContainerStarted","Data":"a4d9438d0e895ed164725a34d5ee4e6343238c4580ea441883afd8664bf2713a"} Oct 30 05:41:17.506604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:17.506422 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:17.546073 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:17.546020 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" podStartSLOduration=1.6102856989999998 podStartE2EDuration="11.546005306s" totalImagesPullingTime="9.935719607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:17.544821212 +0000 UTC m=+637.148510491" watchObservedRunningTime="2026-10-30 05:41:17.546005306 +0000 UTC m=+637.149694562" Oct 30 05:41:28.517406 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:28.517373 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b6gpt" Oct 30 05:41:30.712508 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:41:30.712444 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:41:34.098601 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.098565 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-tttnt"] Oct 30 05:41:34.099080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.098938 5072 cpu_manager.go:414] "containerMap: removing container" podUID="35205cc5-e7a0-4d09-ab49-ba595b9e1fdb" containerName="manager" Oct 30 05:41:34.099080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.098950 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35205cc5-e7a0-4d09-ab49-ba595b9e1fdb" containerName="manager" Oct 30 05:41:34.099080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.099025 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="35205cc5-e7a0-4d09-ab49-ba595b9e1fdb" containerName="manager" Oct 30 05:41:34.226478 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.226432 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-tttnt"] Oct 30 05:41:34.226667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.226561 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.240046 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.240020 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:41:34.250558 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.250523 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:41:34.250719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.250580 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 05:41:34.250719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.250627 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 05:41:34.250719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.250683 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7wbd8\"" Oct 30 05:41:34.315581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.315542 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-metrics-certs\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.315781 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.315602 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-maas-api-tls\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.315781 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.315718 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-openshift-service-ca\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.315781 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.315761 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55fbc\" (UniqueName: \"kubernetes.io/projected/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-kube-api-access-55fbc\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.416962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.416933 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-openshift-service-ca\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.417139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.416978 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55fbc\" (UniqueName: \"kubernetes.io/projected/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-kube-api-access-55fbc\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.417139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.417010 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-metrics-certs\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.417218 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.417188 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-maas-api-tls\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.417649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.417628 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-openshift-service-ca\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.419696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.419671 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-metrics-certs\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.419793 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.419750 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-maas-api-tls\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.444889 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.444866 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55fbc\" (UniqueName: \"kubernetes.io/projected/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-kube-api-access-55fbc\") pod \"maas-api-5c48877c75-tttnt\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.549348 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.549315 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:34.859969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.859941 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-tttnt"] Oct 30 05:41:34.862638 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:41:34.862607 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7765d7_7dd3_43b1_8f9c_4a7d293cc150.slice/crio-6df1c2aa9e214817c278f9278bfcfd36abdfc7f9dcea0e42e00d33ccc118d8c8 WatchSource:0}: Error finding container 6df1c2aa9e214817c278f9278bfcfd36abdfc7f9dcea0e42e00d33ccc118d8c8: Status 404 returned error can't find the container with id 6df1c2aa9e214817c278f9278bfcfd36abdfc7f9dcea0e42e00d33ccc118d8c8 Oct 30 05:41:34.864061 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:34.864046 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:41:35.586623 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:35.586588 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" event={"ID":"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150","Type":"ContainerStarted","Data":"6df1c2aa9e214817c278f9278bfcfd36abdfc7f9dcea0e42e00d33ccc118d8c8"} Oct 30 05:41:37.597544 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:37.597503 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" event={"ID":"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150","Type":"ContainerStarted","Data":"1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75"} Oct 30 05:41:37.597945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:37.597631 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:41:37.639302 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:37.639250 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" podStartSLOduration=1.50427848 podStartE2EDuration="3.639236389s" totalImagesPullingTime="2.134957909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:37.6378453 +0000 UTC m=+657.241534558" watchObservedRunningTime="2026-10-30 05:41:37.639236389 +0000 UTC m=+657.242925646" Oct 30 05:41:43.608363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:41:43.608333 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 05:42:00.709252 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:42:00.709171 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:42:30.709471 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:42:30.709414 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:42:34.852190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.852160 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9"] Oct 30 05:42:34.855944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.855922 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:34.880296 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.880260 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 05:42:34.880296 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.880260 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 05:42:34.880538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.880261 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bgqg5\"" Oct 30 05:42:34.880538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.880260 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 05:42:34.882222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.882197 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9"] Oct 30 05:42:34.949430 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.949387 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:34.949620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.949443 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6vf\" (UniqueName: \"kubernetes.io/projected/df4f6519-1409-4d13-b154-a1313e04c7fd-kube-api-access-5r6vf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:34.949620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.949535 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df4f6519-1409-4d13-b154-a1313e04c7fd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:34.949620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.949583 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:34.949620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.949613 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:34.949791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:34.949718 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.050182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.050143 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.050367 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.050199 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.050367 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.050270 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.050367 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.050329 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.050367 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.050360 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6vf\" (UniqueName: \"kubernetes.io/projected/df4f6519-1409-4d13-b154-a1313e04c7fd-kube-api-access-5r6vf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.050705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.050387 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df4f6519-1409-4d13-b154-a1313e04c7fd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.050705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.050581 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.050705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.050603 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.050878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.050720 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.052738 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.052715 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df4f6519-1409-4d13-b154-a1313e04c7fd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.053070 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.053051 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df4f6519-1409-4d13-b154-a1313e04c7fd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.073479 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.073443 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6vf\" (UniqueName: \"kubernetes.io/projected/df4f6519-1409-4d13-b154-a1313e04c7fd-kube-api-access-5r6vf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jvmq9\" (UID: \"df4f6519-1409-4d13-b154-a1313e04c7fd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.170602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.170569 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:35.472754 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.472727 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9"] Oct 30 05:42:35.474901 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:42:35.474875 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4f6519_1409_4d13_b154_a1313e04c7fd.slice/crio-65cc3c697b700542c18b7abca6c9c74b239d42a743eec27b8e4f77a7405cdce0 WatchSource:0}: Error finding container 65cc3c697b700542c18b7abca6c9c74b239d42a743eec27b8e4f77a7405cdce0: Status 404 returned error can't find the container with id 65cc3c697b700542c18b7abca6c9c74b239d42a743eec27b8e4f77a7405cdce0 Oct 30 05:42:35.875291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:35.875256 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" event={"ID":"df4f6519-1409-4d13-b154-a1313e04c7fd","Type":"ContainerStarted","Data":"65cc3c697b700542c18b7abca6c9c74b239d42a743eec27b8e4f77a7405cdce0"} Oct 30 05:42:36.383598 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.383525 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n"] Oct 30 05:42:36.387996 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.387974 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.416116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.416084 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n"] Oct 30 05:42:36.420167 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.420143 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 05:42:36.462625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.462590 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.462807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.462635 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkpvt\" (UniqueName: \"kubernetes.io/projected/36b1cfca-d939-4a07-92cc-b5bc2520337b-kube-api-access-hkpvt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.462807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.462663 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.462807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.462776 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.462925 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.462835 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.462925 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.462889 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b1cfca-d939-4a07-92cc-b5bc2520337b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.563328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.563283 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.563545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.563350 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.563545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.563375 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.563545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.563419 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b1cfca-d939-4a07-92cc-b5bc2520337b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.563545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.563473 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.563545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.563498 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkpvt\" (UniqueName: \"kubernetes.io/projected/36b1cfca-d939-4a07-92cc-b5bc2520337b-kube-api-access-hkpvt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.564083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.564064 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.564306 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.564287 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.564533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.564504 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.567223 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.567200 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b1cfca-d939-4a07-92cc-b5bc2520337b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.567340 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.567316 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b1cfca-d939-4a07-92cc-b5bc2520337b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.586139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.586110 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkpvt\" (UniqueName: \"kubernetes.io/projected/36b1cfca-d939-4a07-92cc-b5bc2520337b-kube-api-access-hkpvt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7r6n\" (UID: \"36b1cfca-d939-4a07-92cc-b5bc2520337b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.705186 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.705103 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:36.882044 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.882013 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" event={"ID":"df4f6519-1409-4d13-b154-a1313e04c7fd","Type":"ContainerStarted","Data":"79e0dcd35be49d3622307f72dccd55036ac8e9b2cadd66cc17150e8491e84292"} Oct 30 05:42:36.922411 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:36.922357 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" podStartSLOduration=1.666233047 podStartE2EDuration="2.922339653s" totalImagesPullingTime="1.256106606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:36.920830346 +0000 UTC m=+716.524519603" watchObservedRunningTime="2026-10-30 05:42:36.922339653 +0000 UTC m=+716.526028900" Oct 30 05:42:37.029984 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:37.029914 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n"] Oct 30 05:42:37.033925 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:42:37.033895 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b1cfca_d939_4a07_92cc_b5bc2520337b.slice/crio-32f1ea40f3d7be53c67bc2bb04b7aa428ccf2fb43bfbcaf24872d1cb249149f3 WatchSource:0}: Error finding container 32f1ea40f3d7be53c67bc2bb04b7aa428ccf2fb43bfbcaf24872d1cb249149f3: Status 404 returned error can't find the container with id 32f1ea40f3d7be53c67bc2bb04b7aa428ccf2fb43bfbcaf24872d1cb249149f3 Oct 30 05:42:37.170659 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:37.170631 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:37.888208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:37.888170 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" event={"ID":"36b1cfca-d939-4a07-92cc-b5bc2520337b","Type":"ContainerStarted","Data":"4e1a07163d4f137c7492d997c11d816dc237140847e1e0c6da55f0cc683bb94c"} Oct 30 05:42:37.888637 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:37.888219 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" event={"ID":"36b1cfca-d939-4a07-92cc-b5bc2520337b","Type":"ContainerStarted","Data":"32f1ea40f3d7be53c67bc2bb04b7aa428ccf2fb43bfbcaf24872d1cb249149f3"} Oct 30 05:42:37.930564 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:37.930514 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" podStartSLOduration=1.745588049 podStartE2EDuration="1.93049964s" totalImagesPullingTime="184.911591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:37.929834337 +0000 UTC m=+717.533523595" watchObservedRunningTime="2026-10-30 05:42:37.93049964 +0000 UTC m=+717.534188896" Oct 30 05:42:38.706182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:38.706149 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:40.519272 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.519234 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc"] Oct 30 05:42:40.523399 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.523370 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.550085 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.549996 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 05:42:40.552576 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.552549 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc"] Oct 30 05:42:40.603698 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.603665 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpz4\" (UniqueName: \"kubernetes.io/projected/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-kube-api-access-dtpz4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.603884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.603734 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.603884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.603803 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.604020 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.603932 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.604020 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.603958 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.604158 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.604137 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.705051 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.705003 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpz4\" (UniqueName: \"kubernetes.io/projected/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-kube-api-access-dtpz4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.705272 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.705073 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.705272 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.705123 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.705272 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.705190 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.705272 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.705226 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.705590 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.705280 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.705727 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.705691 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.705789 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.705740 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.705927 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.705907 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.707814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.707791 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.708009 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.707992 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.727860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.727838 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpz4\" (UniqueName: \"kubernetes.io/projected/3ceabc67-51ef-4f1c-86dd-88fb31892d7f-kube-api-access-dtpz4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc\" (UID: \"3ceabc67-51ef-4f1c-86dd-88fb31892d7f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:40.837091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:40.837020 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:41.136183 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:41.135982 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc"] Oct 30 05:42:41.138473 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:42:41.138427 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ceabc67_51ef_4f1c_86dd_88fb31892d7f.slice/crio-ebeb8563cfa8924c4ff11f9ea39ea789455db35940d742afa9e41dd3e8dcc630 WatchSource:0}: Error finding container ebeb8563cfa8924c4ff11f9ea39ea789455db35940d742afa9e41dd3e8dcc630: Status 404 returned error can't find the container with id ebeb8563cfa8924c4ff11f9ea39ea789455db35940d742afa9e41dd3e8dcc630 Oct 30 05:42:41.912956 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:41.912918 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" event={"ID":"3ceabc67-51ef-4f1c-86dd-88fb31892d7f","Type":"ContainerStarted","Data":"c7831921cbc8487dcc844c81bc9df11a8a6ac8ba607a0b4179a6c1d782764dac"} Oct 30 05:42:41.913343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:41.912982 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" event={"ID":"3ceabc67-51ef-4f1c-86dd-88fb31892d7f","Type":"ContainerStarted","Data":"ebeb8563cfa8924c4ff11f9ea39ea789455db35940d742afa9e41dd3e8dcc630"} Oct 30 05:42:41.955190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:41.955143 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" podStartSLOduration=1.77667599 podStartE2EDuration="1.955124733s" totalImagesPullingTime="178.448743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:41.954658479 +0000 UTC m=+721.558347737" watchObservedRunningTime="2026-10-30 05:42:41.955124733 +0000 UTC m=+721.558813989" Oct 30 05:42:42.837238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:42.837206 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:45.171527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.171488 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:45.184297 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.184262 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:45.420939 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.420906 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s"] Oct 30 05:42:45.425248 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.425155 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.443717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.443689 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 05:42:45.446569 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.446544 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s"] Oct 30 05:42:45.449569 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.449539 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.449678 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.449618 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhrp\" (UniqueName: \"kubernetes.io/projected/3eb1ca35-0683-4a76-ac18-5fd88ae57463-kube-api-access-vwhrp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.449678 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.449651 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.449772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.449708 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb1ca35-0683-4a76-ac18-5fd88ae57463-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.449772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.449736 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.449836 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.449771 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.551151 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.551117 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.551332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.551171 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhrp\" (UniqueName: \"kubernetes.io/projected/3eb1ca35-0683-4a76-ac18-5fd88ae57463-kube-api-access-vwhrp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.551332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.551192 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.551332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.551225 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb1ca35-0683-4a76-ac18-5fd88ae57463-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.551332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.551243 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.551332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.551265 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.551698 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.551666 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.551797 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.551693 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.551797 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.551753 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.553885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.553857 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb1ca35-0683-4a76-ac18-5fd88ae57463-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.553994 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.553890 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb1ca35-0683-4a76-ac18-5fd88ae57463-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.573007 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.572986 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhrp\" (UniqueName: \"kubernetes.io/projected/3eb1ca35-0683-4a76-ac18-5fd88ae57463-kube-api-access-vwhrp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s\" (UID: \"3eb1ca35-0683-4a76-ac18-5fd88ae57463\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.744225 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.744137 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:45.946847 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:45.946817 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" Oct 30 05:42:46.051245 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:46.051215 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s"] Oct 30 05:42:46.053790 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:42:46.053764 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb1ca35_0683_4a76_ac18_5fd88ae57463.slice/crio-09f66d30fba58e050836c7768e9ae7262ce25dc93c58b408cf70f4a6fe3635d4 WatchSource:0}: Error finding container 09f66d30fba58e050836c7768e9ae7262ce25dc93c58b408cf70f4a6fe3635d4: Status 404 returned error can't find the container with id 09f66d30fba58e050836c7768e9ae7262ce25dc93c58b408cf70f4a6fe3635d4 Oct 30 05:42:46.706066 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:46.706034 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:46.718704 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:46.718682 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:46.939506 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:46.939451 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" event={"ID":"3eb1ca35-0683-4a76-ac18-5fd88ae57463","Type":"ContainerStarted","Data":"124698e2c549c0e0bd559e9bf4c9377f713a739658ad75ef78992687caaeb2c5"} Oct 30 05:42:46.939678 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:46.939536 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" event={"ID":"3eb1ca35-0683-4a76-ac18-5fd88ae57463","Type":"ContainerStarted","Data":"09f66d30fba58e050836c7768e9ae7262ce25dc93c58b408cf70f4a6fe3635d4"} Oct 30 05:42:46.951596 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:46.951571 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" Oct 30 05:42:46.980890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:46.980799 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" podStartSLOduration=1.781878313 podStartE2EDuration="1.980784402s" totalImagesPullingTime="198.906089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:46.979378604 +0000 UTC m=+726.583067861" watchObservedRunningTime="2026-10-30 05:42:46.980784402 +0000 UTC m=+726.584473665" Oct 30 05:42:47.745141 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:47.745106 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:50.837303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:50.837266 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:50.850078 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:50.850049 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:50.974137 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:50.974108 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" Oct 30 05:42:55.744473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:55.744422 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:55.757169 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:55.757144 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:55.999218 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:55.999139 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s" Oct 30 05:42:59.333748 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.333712 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7"] Oct 30 05:42:59.338768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.338741 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:42:59.354559 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.354527 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 05:42:59.354710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.354626 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-gqm89\"" Oct 30 05:42:59.365178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.365154 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 05:42:59.368579 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.368555 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7"] Oct 30 05:42:59.489290 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.489252 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68fpm\" (UniqueName: \"kubernetes.io/projected/6671a91e-715b-44ab-8784-0b806ab553ac-kube-api-access-68fpm\") pod \"ai-gateway-controller-6cf7bcbfb6-2jvr7\" (UID: \"6671a91e-715b-44ab-8784-0b806ab553ac\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:42:59.489501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.489401 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6671a91e-715b-44ab-8784-0b806ab553ac-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-2jvr7\" (UID: \"6671a91e-715b-44ab-8784-0b806ab553ac\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:42:59.590810 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.590725 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6671a91e-715b-44ab-8784-0b806ab553ac-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-2jvr7\" (UID: \"6671a91e-715b-44ab-8784-0b806ab553ac\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:42:59.590810 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.590780 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68fpm\" (UniqueName: \"kubernetes.io/projected/6671a91e-715b-44ab-8784-0b806ab553ac-kube-api-access-68fpm\") pod \"ai-gateway-controller-6cf7bcbfb6-2jvr7\" (UID: \"6671a91e-715b-44ab-8784-0b806ab553ac\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:42:59.593388 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.593359 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6671a91e-715b-44ab-8784-0b806ab553ac-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-2jvr7\" (UID: \"6671a91e-715b-44ab-8784-0b806ab553ac\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:42:59.613034 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.612997 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68fpm\" (UniqueName: \"kubernetes.io/projected/6671a91e-715b-44ab-8784-0b806ab553ac-kube-api-access-68fpm\") pod \"ai-gateway-controller-6cf7bcbfb6-2jvr7\" (UID: \"6671a91e-715b-44ab-8784-0b806ab553ac\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:42:59.652473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.652431 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:42:59.949558 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:42:59.949535 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7"] Oct 30 05:42:59.951863 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:42:59.951821 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6671a91e_715b_44ab_8784_0b806ab553ac.slice/crio-3744f7513508d1c97325c6b9583194e0bca28c62baf4465c03322427d35c7429 WatchSource:0}: Error finding container 3744f7513508d1c97325c6b9583194e0bca28c62baf4465c03322427d35c7429: Status 404 returned error can't find the container with id 3744f7513508d1c97325c6b9583194e0bca28c62baf4465c03322427d35c7429 Oct 30 05:43:00.011604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:00.011569 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" event={"ID":"6671a91e-715b-44ab-8784-0b806ab553ac","Type":"ContainerStarted","Data":"3744f7513508d1c97325c6b9583194e0bca28c62baf4465c03322427d35c7429"} Oct 30 05:43:00.713932 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:43:00.713889 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:43:02.713583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.713557 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv"] Oct 30 05:43:02.718619 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.718601 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.734555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.734521 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf"] Oct 30 05:43:02.738577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.738560 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.739358 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.739343 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-77wcn\"" Oct 30 05:43:02.749150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.749128 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv"] Oct 30 05:43:02.776411 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.776302 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf"] Oct 30 05:43:02.822865 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.822839 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5710a2-f577-4d8a-b42e-48dbf864430f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.822952 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.822881 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.822952 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.822900 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.822952 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.822921 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.823113 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823038 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.823181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823156 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.823227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823210 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.823268 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823242 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t6v9\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-kube-api-access-2t6v9\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.823308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823296 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.823347 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823333 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.823445 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823427 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.823553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823537 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052f4db2-8671-4a39-aa23-cdd244c4288e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.823596 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823586 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.823699 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823683 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.823754 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823713 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.823809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823765 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.823866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823838 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.823913 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.823875 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhmsl\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-kube-api-access-zhmsl\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925228 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925187 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.925428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925243 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925271 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.925428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925294 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925325 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925352 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhmsl\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-kube-api-access-zhmsl\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925389 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5710a2-f577-4d8a-b42e-48dbf864430f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.925428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925422 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925449 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925499 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925529 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925568 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925594 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925625 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t6v9\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-kube-api-access-2t6v9\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925660 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925701 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925703 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.925807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925783 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.926506 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.925834 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052f4db2-8671-4a39-aa23-cdd244c4288e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.926579 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.926540 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052f4db2-8671-4a39-aa23-cdd244c4288e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.926855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.926829 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.927054 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.927028 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.927333 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.927303 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.927630 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.927605 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.927900 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.927755 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5710a2-f577-4d8a-b42e-48dbf864430f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.927900 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.927802 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.928139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.928117 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.928582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.928553 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.928776 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.928209 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.928776 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.928246 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.929503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.929478 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.930219 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.930199 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.957892 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.957788 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhmsl\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-kube-api-access-zhmsl\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.958942 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.958883 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t6v9\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-kube-api-access-2t6v9\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:02.960909 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.960881 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-stjqv\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:02.961495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:02.961446 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-kz9qf\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:03.031718 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:03.031686 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" event={"ID":"6671a91e-715b-44ab-8784-0b806ab553ac","Type":"ContainerStarted","Data":"ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e"} Oct 30 05:43:03.031925 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:03.031900 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:43:03.074174 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:03.074124 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" podStartSLOduration=1.323533571 podStartE2EDuration="4.074106291s" totalImagesPullingTime="2.75057272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:03.071911994 +0000 UTC m=+742.675601251" watchObservedRunningTime="2026-10-30 05:43:03.074106291 +0000 UTC m=+742.677795587" Oct 30 05:43:03.101444 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:03.101407 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:03.120858 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:03.120830 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:03.434202 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:03.434168 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv"] Oct 30 05:43:03.442280 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:43:03.442255 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052f4db2_8671_4a39_aa23_cdd244c4288e.slice/crio-ad563740a9f143dd14ff574919312010b01ae309dcc9641fd78400572f700652 WatchSource:0}: Error finding container ad563740a9f143dd14ff574919312010b01ae309dcc9641fd78400572f700652: Status 404 returned error can't find the container with id ad563740a9f143dd14ff574919312010b01ae309dcc9641fd78400572f700652 Oct 30 05:43:03.455365 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:03.455338 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf"] Oct 30 05:43:03.456535 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:43:03.456508 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5710a2_f577_4d8a_b42e_48dbf864430f.slice/crio-0db6bdaf88b607fc50e6c054690616d0ecbeeb0a2136bc4cdfeea86ef527b342 WatchSource:0}: Error finding container 0db6bdaf88b607fc50e6c054690616d0ecbeeb0a2136bc4cdfeea86ef527b342: Status 404 returned error can't find the container with id 0db6bdaf88b607fc50e6c054690616d0ecbeeb0a2136bc4cdfeea86ef527b342 Oct 30 05:43:04.039875 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.039842 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" event={"ID":"ba5710a2-f577-4d8a-b42e-48dbf864430f","Type":"ContainerStarted","Data":"0db6bdaf88b607fc50e6c054690616d0ecbeeb0a2136bc4cdfeea86ef527b342"} Oct 30 05:43:04.041523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.041487 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" event={"ID":"052f4db2-8671-4a39-aa23-cdd244c4288e","Type":"ContainerStarted","Data":"ad563740a9f143dd14ff574919312010b01ae309dcc9641fd78400572f700652"} Oct 30 05:43:04.175118 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.175085 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xphj8"] Oct 30 05:43:04.180220 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.180197 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:04.205431 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.205394 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dh2m4\"" Oct 30 05:43:04.207407 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.207384 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xphj8"] Oct 30 05:43:04.231758 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.231724 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 05:43:04.340392 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.340309 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcdxz\" (UniqueName: \"kubernetes.io/projected/94eb79a6-731e-4990-abfc-3c9823d595fe-kube-api-access-lcdxz\") pod \"payload-pre-processing-7b98bb678c-xphj8\" (UID: \"94eb79a6-731e-4990-abfc-3c9823d595fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:04.340579 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.340542 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb79a6-731e-4990-abfc-3c9823d595fe-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xphj8\" (UID: \"94eb79a6-731e-4990-abfc-3c9823d595fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:04.441554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.441516 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdxz\" (UniqueName: \"kubernetes.io/projected/94eb79a6-731e-4990-abfc-3c9823d595fe-kube-api-access-lcdxz\") pod \"payload-pre-processing-7b98bb678c-xphj8\" (UID: \"94eb79a6-731e-4990-abfc-3c9823d595fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:04.441766 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.441640 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb79a6-731e-4990-abfc-3c9823d595fe-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xphj8\" (UID: \"94eb79a6-731e-4990-abfc-3c9823d595fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:04.442416 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.442393 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb79a6-731e-4990-abfc-3c9823d595fe-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xphj8\" (UID: \"94eb79a6-731e-4990-abfc-3c9823d595fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:04.465319 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.465285 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdxz\" (UniqueName: \"kubernetes.io/projected/94eb79a6-731e-4990-abfc-3c9823d595fe-kube-api-access-lcdxz\") pod \"payload-pre-processing-7b98bb678c-xphj8\" (UID: \"94eb79a6-731e-4990-abfc-3c9823d595fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:04.494068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.494024 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:04.817397 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:04.817357 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xphj8"] Oct 30 05:43:04.836409 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:43:04.823552 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94eb79a6_731e_4990_abfc_3c9823d595fe.slice/crio-274197c134af6386eb53e79e8cdd1f2829d055b6c5f6053ad421d803cf9de881 WatchSource:0}: Error finding container 274197c134af6386eb53e79e8cdd1f2829d055b6c5f6053ad421d803cf9de881: Status 404 returned error can't find the container with id 274197c134af6386eb53e79e8cdd1f2829d055b6c5f6053ad421d803cf9de881 Oct 30 05:43:05.050517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:05.050442 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" event={"ID":"94eb79a6-731e-4990-abfc-3c9823d595fe","Type":"ContainerStarted","Data":"274197c134af6386eb53e79e8cdd1f2829d055b6c5f6053ad421d803cf9de881"} Oct 30 05:43:07.280574 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:07.280537 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:43:07.280999 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:07.280616 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:43:07.280999 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:07.280646 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:43:07.413088 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:07.413041 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:43:07.413193 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:07.413144 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:43:07.413236 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:07.413193 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:43:08.077014 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.076972 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" event={"ID":"ba5710a2-f577-4d8a-b42e-48dbf864430f","Type":"ContainerStarted","Data":"e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876"} Oct 30 05:43:08.079181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.079149 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" event={"ID":"052f4db2-8671-4a39-aa23-cdd244c4288e","Type":"ContainerStarted","Data":"e82d050862f95b8bd0df1612d8b818e04bb147f679b32b2269193ad700d98bca"} Oct 30 05:43:08.080677 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.080654 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" event={"ID":"94eb79a6-731e-4990-abfc-3c9823d595fe","Type":"ContainerStarted","Data":"51a66a338cd0b234fdd1d48809396dd8bdebe5fae122c644e2feabfe8ae47949"} Oct 30 05:43:08.080781 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.080692 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:08.102043 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.102025 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:08.103812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.103794 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-stjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 05:43:08.103884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.103833 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 05:43:08.125480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.122834 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:08.125480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.122922 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-kz9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 05:43:08.125480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.122971 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 05:43:08.193311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.193247 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" podStartSLOduration=2.225616707 podStartE2EDuration="6.193229314s" totalImagesPullingTime="3.967612607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:08.192236989 +0000 UTC m=+747.795926245" watchObservedRunningTime="2026-10-30 05:43:08.193229314 +0000 UTC m=+747.796918572" Oct 30 05:43:08.193500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:08.193431 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podStartSLOduration=2.372351086 podStartE2EDuration="6.193422287s" totalImagesPullingTime="3.821071201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:08.137151618 +0000 UTC m=+747.740840879" watchObservedRunningTime="2026-10-30 05:43:08.193422287 +0000 UTC m=+747.797111544" Oct 30 05:43:09.101942 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:09.101910 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-stjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 05:43:09.102337 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:09.101966 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 05:43:09.121442 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:09.121316 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-kz9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 05:43:09.121442 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:09.121382 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 05:43:10.101918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:10.101882 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-stjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 05:43:10.102095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:10.101936 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 05:43:10.132485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:10.129078 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-kz9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 05:43:10.132485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:10.129133 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 05:43:11.102422 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:11.102388 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-stjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 05:43:11.102830 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:11.102449 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 05:43:11.129484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:11.125564 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-kz9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 05:43:11.129484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:11.125632 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 05:43:12.102377 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:12.102340 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-stjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 05:43:12.102609 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:12.102398 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 05:43:12.132485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:12.132241 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-kz9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 05:43:12.132485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:12.132333 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 05:43:13.101866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:13.101834 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-stjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 05:43:13.102052 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:13.101889 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 05:43:13.128484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:13.125851 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-kz9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 05:43:13.128484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:13.125917 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 05:43:14.044805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:14.044766 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:43:14.083071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:14.083014 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" podStartSLOduration=7.494145562 podStartE2EDuration="10.083000565s" totalImagesPullingTime="2.588855003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:08.23423857 +0000 UTC m=+747.837927828" watchObservedRunningTime="2026-10-30 05:43:14.083000565 +0000 UTC m=+753.686689824" Oct 30 05:43:14.102418 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:14.102388 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-stjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 05:43:14.102655 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:14.102495 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 05:43:14.121735 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:14.121700 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-kz9qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 05:43:14.121915 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:14.121752 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 05:43:15.735261 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:15.735229 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:15.736530 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:15.736502 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:15.737278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:15.737259 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 05:43:15.866923 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:15.866882 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:15.868237 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:15.868214 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:15.869151 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:15.869122 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:43:17.779134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.779099 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw"] Oct 30 05:43:17.784612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.784588 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.814901 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.814868 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw"] Oct 30 05:43:17.829018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.828989 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.829018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.829021 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.829276 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.829101 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.829276 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.829132 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.829276 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.829200 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.829504 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.829311 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.829504 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.829340 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.829504 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.829362 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl465\" (UniqueName: \"kubernetes.io/projected/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-kube-api-access-wl465\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.829504 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.829388 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.930756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.930724 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.930756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.930761 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.930975 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.930807 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931028 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.930973 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931028 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931076 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931116 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931147 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl465\" (UniqueName: \"kubernetes.io/projected/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-kube-api-access-wl465\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931184 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931491 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931560 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931654 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.931918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931802 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.932009 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.931984 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.933614 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.933598 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.933683 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.933644 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.952687 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.952661 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:17.953164 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:17.953145 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl465\" (UniqueName: \"kubernetes.io/projected/6b24ad0f-1934-49a2-9da3-95b2c8769b2f-kube-api-access-wl465\") pod \"maas-default-gateway-openshift-default-84587756dc-ctspw\" (UID: \"6b24ad0f-1934-49a2-9da3-95b2c8769b2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:18.099017 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:18.098940 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:18.416344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:18.416319 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw"] Oct 30 05:43:18.417951 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:43:18.417929 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b24ad0f_1934_49a2_9da3_95b2c8769b2f.slice/crio-501b8903142dd689e6990648743b3341b8613b4602da0ac72020cd4507785ca4 WatchSource:0}: Error finding container 501b8903142dd689e6990648743b3341b8613b4602da0ac72020cd4507785ca4: Status 404 returned error can't find the container with id 501b8903142dd689e6990648743b3341b8613b4602da0ac72020cd4507785ca4 Oct 30 05:43:18.420236 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:18.420208 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:43:18.420326 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:18.420274 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:43:18.420326 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:18.420306 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:43:19.088600 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:19.088569 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xphj8" Oct 30 05:43:19.142672 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:19.142633 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" event={"ID":"6b24ad0f-1934-49a2-9da3-95b2c8769b2f","Type":"ContainerStarted","Data":"4707f55763771ca8606728efa4d242b40bddfaa6832355f593465f9eed40dfb3"} Oct 30 05:43:19.142843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:19.142691 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" event={"ID":"6b24ad0f-1934-49a2-9da3-95b2c8769b2f","Type":"ContainerStarted","Data":"501b8903142dd689e6990648743b3341b8613b4602da0ac72020cd4507785ca4"} Oct 30 05:43:19.188067 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:19.188006 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" podStartSLOduration=2.187986771 podStartE2EDuration="2.187986771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:19.186209535 +0000 UTC m=+758.789898816" watchObservedRunningTime="2026-10-30 05:43:19.187986771 +0000 UTC m=+758.791676029" Oct 30 05:43:20.099960 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:20.099929 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:20.101467 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:20.101432 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ctspw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:43:20.101531 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:20.101501 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" podUID="6b24ad0f-1934-49a2-9da3-95b2c8769b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 05:43:21.099757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:21.099723 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ctspw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:43:21.099930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:21.099774 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" podUID="6b24ad0f-1934-49a2-9da3-95b2c8769b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 05:43:22.100308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:22.100274 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ctspw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:43:22.100724 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:22.100324 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" podUID="6b24ad0f-1934-49a2-9da3-95b2c8769b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 05:43:23.099294 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:23.099260 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ctspw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:43:23.099499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:23.099321 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" podUID="6b24ad0f-1934-49a2-9da3-95b2c8769b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 05:43:24.099604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:24.099567 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ctspw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:43:24.100093 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:24.099621 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" podUID="6b24ad0f-1934-49a2-9da3-95b2c8769b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 05:43:25.099538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:25.099503 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ctspw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:43:25.099728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:25.099562 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" podUID="6b24ad0f-1934-49a2-9da3-95b2c8769b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 05:43:26.662291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:26.662262 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:26.663053 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:26.663023 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:26.663915 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:26.663896 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" Oct 30 05:43:28.413551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:28.413502 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7"] Oct 30 05:43:28.414017 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:28.413722 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" podUID="6671a91e-715b-44ab-8784-0b806ab553ac" containerName="manager" containerID="cri-o://ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e" gracePeriod=30 Oct 30 05:43:28.672188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:28.672135 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:43:28.734385 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:28.734342 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6671a91e-715b-44ab-8784-0b806ab553ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6671a91e-715b-44ab-8784-0b806ab553ac-metrics-certs\") pod \"6671a91e-715b-44ab-8784-0b806ab553ac\" (UID: \"6671a91e-715b-44ab-8784-0b806ab553ac\") " Oct 30 05:43:28.734580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:28.734486 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6671a91e-715b-44ab-8784-0b806ab553ac-kube-api-access-68fpm\" (UniqueName: \"kubernetes.io/projected/6671a91e-715b-44ab-8784-0b806ab553ac-kube-api-access-68fpm\") pod \"6671a91e-715b-44ab-8784-0b806ab553ac\" (UID: \"6671a91e-715b-44ab-8784-0b806ab553ac\") " Oct 30 05:43:28.736603 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:28.736572 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6671a91e-715b-44ab-8784-0b806ab553ac-metrics-certs" pod "6671a91e-715b-44ab-8784-0b806ab553ac" (UID: "6671a91e-715b-44ab-8784-0b806ab553ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:43:28.736717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:28.736633 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6671a91e-715b-44ab-8784-0b806ab553ac-kube-api-access-68fpm" pod "6671a91e-715b-44ab-8784-0b806ab553ac" (UID: "6671a91e-715b-44ab-8784-0b806ab553ac"). InnerVolumeSpecName "kube-api-access-68fpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:43:28.835407 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:28.835371 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6671a91e-715b-44ab-8784-0b806ab553ac-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:43:28.835407 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:28.835401 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68fpm\" (UniqueName: \"kubernetes.io/projected/6671a91e-715b-44ab-8784-0b806ab553ac-kube-api-access-68fpm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:43:29.196512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:29.196441 5072 generic.go:356] "Generic (PLEG): container finished" podID="6671a91e-715b-44ab-8784-0b806ab553ac" containerID="ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e" exitCode=0 Oct 30 05:43:29.196512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:29.196504 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" Oct 30 05:43:29.196726 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:29.196531 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" event={"ID":"6671a91e-715b-44ab-8784-0b806ab553ac","Type":"ContainerDied","Data":"ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e"} Oct 30 05:43:29.196726 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:29.196586 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7" event={"ID":"6671a91e-715b-44ab-8784-0b806ab553ac","Type":"ContainerDied","Data":"3744f7513508d1c97325c6b9583194e0bca28c62baf4465c03322427d35c7429"} Oct 30 05:43:29.196726 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:29.196610 5072 scope.go:122] "RemoveContainer" containerID="ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e" Oct 30 05:43:29.205752 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:29.205732 5072 scope.go:122] "RemoveContainer" containerID="ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e" Oct 30 05:43:29.205999 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:43:29.205978 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e\": container with ID starting with ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e not found: ID does not exist" containerID="ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e" Oct 30 05:43:29.206064 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:29.206012 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e"} err="failed to get container status \"ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e\": rpc error: code = NotFound desc = could not find container \"ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e\": container with ID starting with ad3ac1bf2191ac9b1b6bff6716e64d661c096146646b6c0fc243dfa782790b7e not found: ID does not exist" Oct 30 05:43:29.244194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:29.244163 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7"] Oct 30 05:43:29.254827 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:29.254800 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-2jvr7"] Oct 30 05:43:30.516535 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:43:30.516504 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6671a91e-715b-44ab-8784-0b806ab553ac" path="/var/lib/kubelet/pods/6671a91e-715b-44ab-8784-0b806ab553ac/volumes" Oct 30 05:43:30.708678 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:43:30.708646 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:44:00.711134 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:44:00.711101 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:44:00.926004 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:00.925975 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t"] Oct 30 05:44:00.926514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:00.926496 5072 cpu_manager.go:414] "containerMap: removing container" podUID="6671a91e-715b-44ab-8784-0b806ab553ac" containerName="manager" Oct 30 05:44:00.926619 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:00.926515 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6671a91e-715b-44ab-8784-0b806ab553ac" containerName="manager" Oct 30 05:44:00.926677 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:00.926664 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6671a91e-715b-44ab-8784-0b806ab553ac" containerName="manager" Oct 30 05:44:00.929814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:00.929792 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:00.960816 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:00.960791 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1d8c2f-openshift-default-dockercfg-7kqf6\"" Oct 30 05:44:00.969203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:00.969122 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t"] Oct 30 05:44:01.051883 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.051846 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.052082 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.051912 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.052082 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.052014 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxpt\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-kube-api-access-bgxpt\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.052082 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.052056 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.052291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.052152 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.052291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.052182 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.052291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.052219 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.052453 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.052339 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.052453 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.052383 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.153604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.153564 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.153820 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.153651 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.153820 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.153746 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxpt\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-kube-api-access-bgxpt\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.153820 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.153780 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.154019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.153820 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.154019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.153843 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.154019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.153875 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.154019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.153918 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.154019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.153959 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.154412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.154390 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.156585 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.156559 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.156966 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.156943 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.157051 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.157009 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.157492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.157472 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.157675 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.157653 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.157762 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.157742 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.157817 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.157762 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc"] Oct 30 05:44:01.162076 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.162061 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.190572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.190549 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-698ab2-openshift-default-dockercfg-fl985\"" Oct 30 05:44:01.204349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.204319 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc"] Oct 30 05:44:01.207169 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.207142 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.207604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.207585 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxpt\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-kube-api-access-bgxpt\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.244577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.244501 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:01.255284 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.255250 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.255472 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.255315 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.255472 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.255338 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.255618 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.255528 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ghr\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-kube-api-access-88ghr\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.255618 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.255573 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.255747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.255657 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.255747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.255704 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ff58d-8474-487f-8bbb-a7d068c56a65-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.255861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.255764 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.255861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.255816 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.357287 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.357251 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.357512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.357308 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.357512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.357361 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ghr\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-kube-api-access-88ghr\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.357512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.357416 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.357512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.357498 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.357747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.357544 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ff58d-8474-487f-8bbb-a7d068c56a65-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.357747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.357582 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.357747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.357673 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.357958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.357748 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.359174 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.359114 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.360339 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.359766 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.360339 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.358960 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.360339 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.359978 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.360339 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.360215 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ff58d-8474-487f-8bbb-a7d068c56a65-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.362690 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.362664 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.363157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.363138 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.390804 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.390777 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.390950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.390939 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ghr\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-kube-api-access-88ghr\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.486378 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.486320 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:01.620415 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.620389 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t"] Oct 30 05:44:01.622547 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:44:01.622521 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e940ff_2f2d_4c6d_bc35_84d57448ddf2.slice/crio-89f321a3753ad713d86c6f86f188f034db02acce4105e1f7b5f7cbea9b5e44e8 WatchSource:0}: Error finding container 89f321a3753ad713d86c6f86f188f034db02acce4105e1f7b5f7cbea9b5e44e8: Status 404 returned error can't find the container with id 89f321a3753ad713d86c6f86f188f034db02acce4105e1f7b5f7cbea9b5e44e8 Oct 30 05:44:01.624765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.624734 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:01.624844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.624804 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:01.624844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.624829 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:01.697905 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.697805 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2"] Oct 30 05:44:01.702496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.701500 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.725619 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.725591 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e7a506-openshift-default-dockercfg-552kl\"" Oct 30 05:44:01.740254 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.740221 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2"] Oct 30 05:44:01.769104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.766032 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.769104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.766096 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.769104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.766145 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8c667c-d266-42c4-a528-c8957316da9c-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.769104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.766237 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c667c-d266-42c4-a528-c8957316da9c-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.769104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.766267 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.769104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.766307 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.769104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.766571 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdsr\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-kube-api-access-xwdsr\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.769104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.766633 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.769104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.766685 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.811950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.811918 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc"] Oct 30 05:44:01.813490 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:44:01.813434 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494ff58d_8474_487f_8bbb_a7d068c56a65.slice/crio-79a09ad4022290a7568c458d1aa8b012b5310863048206469191cef34fce7f57 WatchSource:0}: Error finding container 79a09ad4022290a7568c458d1aa8b012b5310863048206469191cef34fce7f57: Status 404 returned error can't find the container with id 79a09ad4022290a7568c458d1aa8b012b5310863048206469191cef34fce7f57 Oct 30 05:44:01.816353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.816317 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:01.816437 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.816404 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:01.816498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.816442 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:01.867393 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.867368 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.867557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.867434 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.867557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.867492 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.867557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.867537 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8c667c-d266-42c4-a528-c8957316da9c-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.867740 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.867624 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c667c-d266-42c4-a528-c8957316da9c-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.867740 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.867657 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.867740 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.867700 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.867893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.867756 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdsr\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-kube-api-access-xwdsr\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.867893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.867788 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.868159 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.868138 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.868245 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.868196 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.868402 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.868384 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.868851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.868828 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.869411 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.869393 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c667c-d266-42c4-a528-c8957316da9c-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.870502 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.870451 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.870945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.870914 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8c667c-d266-42c4-a528-c8957316da9c-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.919530 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.916559 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdsr\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-kube-api-access-xwdsr\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:01.927295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:01.927234 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:02.030070 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.028713 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:02.360426 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.360385 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" event={"ID":"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2","Type":"ContainerStarted","Data":"8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8"} Oct 30 05:44:02.360666 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.360451 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" event={"ID":"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2","Type":"ContainerStarted","Data":"89f321a3753ad713d86c6f86f188f034db02acce4105e1f7b5f7cbea9b5e44e8"} Oct 30 05:44:02.362201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.362177 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" event={"ID":"494ff58d-8474-487f-8bbb-a7d068c56a65","Type":"ContainerStarted","Data":"bd2f381c5ef10b2ebafd1b4bab90bc5438331606c11156831f820a2ec146073a"} Oct 30 05:44:02.362338 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.362215 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" event={"ID":"494ff58d-8474-487f-8bbb-a7d068c56a65","Type":"ContainerStarted","Data":"79a09ad4022290a7568c458d1aa8b012b5310863048206469191cef34fce7f57"} Oct 30 05:44:02.385349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.385310 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2"] Oct 30 05:44:02.387895 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:44:02.387854 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8c667c_d266_42c4_a528_c8957316da9c.slice/crio-7d53ea68a210ee122c04036fba35d8283a073403c2700cbf1b8f3a45bb10d77b WatchSource:0}: Error finding container 7d53ea68a210ee122c04036fba35d8283a073403c2700cbf1b8f3a45bb10d77b: Status 404 returned error can't find the container with id 7d53ea68a210ee122c04036fba35d8283a073403c2700cbf1b8f3a45bb10d77b Oct 30 05:44:02.390368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.390314 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:02.390477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.390413 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:02.390525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.390472 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:02.406883 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.406829 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" podStartSLOduration=2.40681062 podStartE2EDuration="2.40681062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:02.405477172 +0000 UTC m=+802.009166429" watchObservedRunningTime="2026-10-30 05:44:02.40681062 +0000 UTC m=+802.010499877" Oct 30 05:44:02.459632 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.459584 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" podStartSLOduration=1.459568571 podStartE2EDuration="1.459568571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:02.457680967 +0000 UTC m=+802.061370253" watchObservedRunningTime="2026-10-30 05:44:02.459568571 +0000 UTC m=+802.063257828" Oct 30 05:44:02.486752 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.486719 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:02.492592 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:02.492566 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:03.245211 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:03.245175 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:03.250329 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:03.250306 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:03.369679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:03.369637 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" event={"ID":"fc8c667c-d266-42c4-a528-c8957316da9c","Type":"ContainerStarted","Data":"dfce374ad48b8e35a7e37e6b884c597910886ede3a60d79bebcfd414a61707be"} Oct 30 05:44:03.369880 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:03.369700 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" event={"ID":"fc8c667c-d266-42c4-a528-c8957316da9c","Type":"ContainerStarted","Data":"7d53ea68a210ee122c04036fba35d8283a073403c2700cbf1b8f3a45bb10d77b"} Oct 30 05:44:03.370241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:03.370133 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:03.370241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:03.370178 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:03.370968 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:03.370949 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:03.371142 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:03.371125 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:03.459321 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:03.459272 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" podStartSLOduration=2.459255431 podStartE2EDuration="2.459255431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:03.456912085 +0000 UTC m=+803.060601355" watchObservedRunningTime="2026-10-30 05:44:03.459255431 +0000 UTC m=+803.062944689" Oct 30 05:44:04.030257 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:04.030213 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:04.035322 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:04.035299 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:04.375420 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:04.375366 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:04.376365 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:04.376343 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:06.636145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.636109 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb"] Oct 30 05:44:06.641656 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.641626 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.658855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.657897 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-698ab2-serving-cert\"" Oct 30 05:44:06.672437 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.672356 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb"] Oct 30 05:44:06.729415 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.729385 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-metrics-certs\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.729603 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.729431 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-maas-api-tls\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.729650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.729622 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922baf8-fe45-4754-a3fd-7d604a4dee19-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.729686 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.729663 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnpzz\" (UniqueName: \"kubernetes.io/projected/b922baf8-fe45-4754-a3fd-7d604a4dee19-kube-api-access-xnpzz\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.831083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.831045 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnpzz\" (UniqueName: \"kubernetes.io/projected/b922baf8-fe45-4754-a3fd-7d604a4dee19-kube-api-access-xnpzz\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.831247 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.831106 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-metrics-certs\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.831247 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.831161 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-maas-api-tls\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.831377 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.831350 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922baf8-fe45-4754-a3fd-7d604a4dee19-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.832429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.832404 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922baf8-fe45-4754-a3fd-7d604a4dee19-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.834755 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.834729 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-metrics-certs\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.834928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.834871 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-maas-api-tls\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.886241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.886167 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnpzz\" (UniqueName: \"kubernetes.io/projected/b922baf8-fe45-4754-a3fd-7d604a4dee19-kube-api-access-xnpzz\") pod \"maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:06.964107 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:06.964073 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:07.314137 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:07.314113 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb"] Oct 30 05:44:07.315292 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:44:07.315266 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb922baf8_fe45_4754_a3fd_7d604a4dee19.slice/crio-86f09591f4a9768edffff3f50ab2fd1b7a33dfb87c787ea2c9bb5b7ca58a9ca9 WatchSource:0}: Error finding container 86f09591f4a9768edffff3f50ab2fd1b7a33dfb87c787ea2c9bb5b7ca58a9ca9: Status 404 returned error can't find the container with id 86f09591f4a9768edffff3f50ab2fd1b7a33dfb87c787ea2c9bb5b7ca58a9ca9 Oct 30 05:44:07.394307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:07.394269 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" event={"ID":"b922baf8-fe45-4754-a3fd-7d604a4dee19","Type":"ContainerStarted","Data":"86f09591f4a9768edffff3f50ab2fd1b7a33dfb87c787ea2c9bb5b7ca58a9ca9"} Oct 30 05:44:08.413045 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:08.413015 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" event={"ID":"b922baf8-fe45-4754-a3fd-7d604a4dee19","Type":"ContainerStarted","Data":"17b67e03869fd47810260bf91d07cd6cb8b6d24832a64eabb5db27e885791b63"} Oct 30 05:44:08.413542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:08.413146 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:08.483522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:08.483443 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" podStartSLOduration=2.125074906 podStartE2EDuration="2.483424601s" totalImagesPullingTime="358.349695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:08.48232784 +0000 UTC m=+808.086017088" watchObservedRunningTime="2026-10-30 05:44:08.483424601 +0000 UTC m=+808.087113859" Oct 30 05:44:09.218110 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.218074 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr"] Oct 30 05:44:09.222782 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.222759 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.236321 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.236300 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-1d8c2f-serving-cert\"" Oct 30 05:44:09.254244 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.254211 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr"] Oct 30 05:44:09.376524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.376488 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17588e96-befd-440e-aca9-e8582a61fcdf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.376696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.376666 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-metrics-certs\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.376747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.376726 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.376818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.376796 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2c6q\" (UniqueName: \"kubernetes.io/projected/17588e96-befd-440e-aca9-e8582a61fcdf-kube-api-access-s2c6q\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.477701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.477618 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-metrics-certs\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.477701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.477665 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.478157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.477847 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2c6q\" (UniqueName: \"kubernetes.io/projected/17588e96-befd-440e-aca9-e8582a61fcdf-kube-api-access-s2c6q\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.478157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.477910 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17588e96-befd-440e-aca9-e8582a61fcdf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.478563 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.478542 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17588e96-befd-440e-aca9-e8582a61fcdf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.480395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.480370 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-metrics-certs\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.480680 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.480662 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.500967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.500930 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2c6q\" (UniqueName: \"kubernetes.io/projected/17588e96-befd-440e-aca9-e8582a61fcdf-kube-api-access-s2c6q\") pod \"maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.538734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.538702 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:09.842792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:09.842764 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr"] Oct 30 05:44:09.844769 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:44:09.844745 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17588e96_befd_440e_aca9_e8582a61fcdf.slice/crio-030f0ba566efca380da7cfdda7f7463f5ecd2e8fdf84cad53a7beb2f2141989f WatchSource:0}: Error finding container 030f0ba566efca380da7cfdda7f7463f5ecd2e8fdf84cad53a7beb2f2141989f: Status 404 returned error can't find the container with id 030f0ba566efca380da7cfdda7f7463f5ecd2e8fdf84cad53a7beb2f2141989f Oct 30 05:44:10.440759 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:10.440721 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" event={"ID":"17588e96-befd-440e-aca9-e8582a61fcdf","Type":"ContainerStarted","Data":"f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8"} Oct 30 05:44:10.440934 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:10.440788 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" event={"ID":"17588e96-befd-440e-aca9-e8582a61fcdf","Type":"ContainerStarted","Data":"030f0ba566efca380da7cfdda7f7463f5ecd2e8fdf84cad53a7beb2f2141989f"} Oct 30 05:44:10.440934 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:10.440833 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:10.482167 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:10.482112 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" podStartSLOduration=1.059582576 podStartE2EDuration="1.482094334s" totalImagesPullingTime="422.511758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:10.480622093 +0000 UTC m=+810.084311362" watchObservedRunningTime="2026-10-30 05:44:10.482094334 +0000 UTC m=+810.085783591" Oct 30 05:44:13.251988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.251954 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb"] Oct 30 05:44:13.257967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.257944 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.276452 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.274982 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-64123f-serving-cert\"" Oct 30 05:44:13.290653 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.290600 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb"] Oct 30 05:44:13.299411 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.299378 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.299609 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.299582 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-metrics-certs\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.299684 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.299655 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416250e1-133e-4f0d-98e9-5d5d6c0f8105-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.299759 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.299719 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmpx4\" (UniqueName: \"kubernetes.io/projected/416250e1-133e-4f0d-98e9-5d5d6c0f8105-kube-api-access-xmpx4\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.401114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.401077 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-metrics-certs\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.401279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.401133 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416250e1-133e-4f0d-98e9-5d5d6c0f8105-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.401279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.401166 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmpx4\" (UniqueName: \"kubernetes.io/projected/416250e1-133e-4f0d-98e9-5d5d6c0f8105-kube-api-access-xmpx4\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.401279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.401198 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.401473 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:44:13.401310 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-serving-cert: secret "maas-api-e2e-worker-w6-64123f-serving-cert" not found Oct 30 05:44:13.401473 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:44:13.401366 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls podName:416250e1-133e-4f0d-98e9-5d5d6c0f8105 nodeName:}" failed. No retries permitted until 2026-10-30 05:44:13.901350785 +0000 UTC m=+813.505040021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls") pod "maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" (UID: "416250e1-133e-4f0d-98e9-5d5d6c0f8105") : secret "maas-api-e2e-worker-w6-64123f-serving-cert" not found Oct 30 05:44:13.402551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.402521 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416250e1-133e-4f0d-98e9-5d5d6c0f8105-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.404625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.404604 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-metrics-certs\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.545548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.524873 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmpx4\" (UniqueName: \"kubernetes.io/projected/416250e1-133e-4f0d-98e9-5d5d6c0f8105-kube-api-access-xmpx4\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.939229 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.939187 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:13.947501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:13.947069 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls\") pod \"maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:14.171738 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:14.171704 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:14.445055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:14.444968 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 05:44:14.552182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:14.552156 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb"] Oct 30 05:44:14.553974 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:44:14.553945 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416250e1_133e_4f0d_98e9_5d5d6c0f8105.slice/crio-db24669af5824db4e381e401c366114cbf480bd0c9fb8b29759564974eb9c2e0 WatchSource:0}: Error finding container db24669af5824db4e381e401c366114cbf480bd0c9fb8b29759564974eb9c2e0: Status 404 returned error can't find the container with id db24669af5824db4e381e401c366114cbf480bd0c9fb8b29759564974eb9c2e0 Oct 30 05:44:15.443485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.442549 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8"] Oct 30 05:44:15.461516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.452774 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.471482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.470533 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8"] Oct 30 05:44:15.471482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.470809 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-be78f2-serving-cert\"" Oct 30 05:44:15.493604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.493570 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-metrics-certs\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.494095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.494069 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.494278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.494255 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4klb\" (UniqueName: \"kubernetes.io/projected/96a7a0ba-240b-45ae-9df6-ef1d323cde72-kube-api-access-h4klb\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.494541 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.494519 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a7a0ba-240b-45ae-9df6-ef1d323cde72-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.584515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.582019 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" event={"ID":"416250e1-133e-4f0d-98e9-5d5d6c0f8105","Type":"ContainerStarted","Data":"e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7"} Oct 30 05:44:15.584515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.582084 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" event={"ID":"416250e1-133e-4f0d-98e9-5d5d6c0f8105","Type":"ContainerStarted","Data":"db24669af5824db4e381e401c366114cbf480bd0c9fb8b29759564974eb9c2e0"} Oct 30 05:44:15.584515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.582133 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:15.595908 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.595879 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-metrics-certs\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.596037 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.595972 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.596037 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.595989 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4klb\" (UniqueName: \"kubernetes.io/projected/96a7a0ba-240b-45ae-9df6-ef1d323cde72-kube-api-access-h4klb\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.596037 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.596035 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a7a0ba-240b-45ae-9df6-ef1d323cde72-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.596727 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:44:15.596707 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-serving-cert: secret "maas-api-e2e-worker-w5-be78f2-serving-cert" not found Oct 30 05:44:15.596892 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:44:15.596881 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls podName:96a7a0ba-240b-45ae-9df6-ef1d323cde72 nodeName:}" failed. No retries permitted until 2026-10-30 05:44:16.096864229 +0000 UTC m=+815.700553489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls") pod "maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" (UID: "96a7a0ba-240b-45ae-9df6-ef1d323cde72") : secret "maas-api-e2e-worker-w5-be78f2-serving-cert" not found Oct 30 05:44:15.597476 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.597437 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a7a0ba-240b-45ae-9df6-ef1d323cde72-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.599283 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.599265 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-metrics-certs\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:15.648486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:15.644640 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4klb\" (UniqueName: \"kubernetes.io/projected/96a7a0ba-240b-45ae-9df6-ef1d323cde72-kube-api-access-h4klb\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:16.105082 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:16.105049 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:16.155103 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:16.155070 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls\") pod \"maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:16.388089 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:16.387982 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:16.478489 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:16.477724 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 05:44:16.557499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:16.550119 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" podStartSLOduration=3.154192853 podStartE2EDuration="3.550099083s" totalImagesPullingTime="395.90623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:15.640011235 +0000 UTC m=+815.243700548" watchObservedRunningTime="2026-10-30 05:44:16.550099083 +0000 UTC m=+816.153788343" Oct 30 05:44:16.779315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:16.779176 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8"] Oct 30 05:44:16.785437 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:44:16.785408 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a7a0ba_240b_45ae_9df6_ef1d323cde72.slice/crio-586c9a69766f1785804c39322e105bf5c3ccbd24cdf30fd7bfbd797d732229fc WatchSource:0}: Error finding container 586c9a69766f1785804c39322e105bf5c3ccbd24cdf30fd7bfbd797d732229fc: Status 404 returned error can't find the container with id 586c9a69766f1785804c39322e105bf5c3ccbd24cdf30fd7bfbd797d732229fc Oct 30 05:44:17.532565 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.529028 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb"] Oct 30 05:44:17.541581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.539180 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.546120 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.546028 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb"] Oct 30 05:44:17.554955 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.554090 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-fffaa8-serving-cert\"" Oct 30 05:44:17.581304 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.581267 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28026c62-1c09-45d4-9d3e-810c67b7117c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.581498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.581358 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-metrics-certs\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.581498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.581418 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.581628 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.581539 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm7rw\" (UniqueName: \"kubernetes.io/projected/28026c62-1c09-45d4-9d3e-810c67b7117c-kube-api-access-gm7rw\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.602245 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.602202 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" event={"ID":"96a7a0ba-240b-45ae-9df6-ef1d323cde72","Type":"ContainerStarted","Data":"bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f"} Oct 30 05:44:17.602404 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.602280 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" event={"ID":"96a7a0ba-240b-45ae-9df6-ef1d323cde72","Type":"ContainerStarted","Data":"586c9a69766f1785804c39322e105bf5c3ccbd24cdf30fd7bfbd797d732229fc"} Oct 30 05:44:17.603307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.603287 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:17.670834 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.670779 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" podStartSLOduration=2.300091211 podStartE2EDuration="2.670764617s" totalImagesPullingTime="370.673406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:17.66918335 +0000 UTC m=+817.272872610" watchObservedRunningTime="2026-10-30 05:44:17.670764617 +0000 UTC m=+817.274453895" Oct 30 05:44:17.682764 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.682736 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm7rw\" (UniqueName: \"kubernetes.io/projected/28026c62-1c09-45d4-9d3e-810c67b7117c-kube-api-access-gm7rw\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.682897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.682836 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28026c62-1c09-45d4-9d3e-810c67b7117c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.682976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.682947 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-metrics-certs\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.683246 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.683217 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.683407 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:44:17.683384 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-serving-cert: secret "maas-api-e2e-worker-w3-fffaa8-serving-cert" not found Oct 30 05:44:17.683504 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:44:17.683471 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls podName:28026c62-1c09-45d4-9d3e-810c67b7117c nodeName:}" failed. No retries permitted until 2026-10-30 05:44:18.183433037 +0000 UTC m=+817.787122274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls") pod "maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" (UID: "28026c62-1c09-45d4-9d3e-810c67b7117c") : secret "maas-api-e2e-worker-w3-fffaa8-serving-cert" not found Oct 30 05:44:17.683597 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.683576 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28026c62-1c09-45d4-9d3e-810c67b7117c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.686328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.686305 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-metrics-certs\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:17.705591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:17.705563 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm7rw\" (UniqueName: \"kubernetes.io/projected/28026c62-1c09-45d4-9d3e-810c67b7117c-kube-api-access-gm7rw\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:18.190012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:18.189951 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:18.197166 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:18.197132 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:18.470978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:18.470897 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:18.841846 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:18.841813 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb"] Oct 30 05:44:19.418487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:19.416590 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:44:19.418487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:19.416658 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:44:19.642415 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:19.642377 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" event={"ID":"28026c62-1c09-45d4-9d3e-810c67b7117c","Type":"ContainerStarted","Data":"9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9"} Oct 30 05:44:19.642613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:19.642435 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" event={"ID":"28026c62-1c09-45d4-9d3e-810c67b7117c","Type":"ContainerStarted","Data":"220eee863ee6cf6d19074c06ad94c3dc3bbb0a2343eae2c9fcf700c5427d978f"} Oct 30 05:44:19.642613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:19.642504 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:19.693228 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:19.693153 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" podStartSLOduration=2.296541843 podStartE2EDuration="2.69313241s" totalImagesPullingTime="396.590567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:19.690247774 +0000 UTC m=+819.293937032" watchObservedRunningTime="2026-10-30 05:44:19.69313241 +0000 UTC m=+819.296821668" Oct 30 05:44:21.597625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:21.597606 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 05:44:21.661612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:21.661432 5072 generic.go:356] "Generic (PLEG): container finished" podID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerID="bd2f381c5ef10b2ebafd1b4bab90bc5438331606c11156831f820a2ec146073a" exitCode=0 Oct 30 05:44:21.661849 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:21.661516 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" event={"ID":"494ff58d-8474-487f-8bbb-a7d068c56a65","Type":"ContainerDied","Data":"bd2f381c5ef10b2ebafd1b4bab90bc5438331606c11156831f820a2ec146073a"} Oct 30 05:44:21.662509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:21.662487 5072 scope.go:122] "RemoveContainer" containerID="bd2f381c5ef10b2ebafd1b4bab90bc5438331606c11156831f820a2ec146073a" Oct 30 05:44:21.665027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:21.664987 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:21.665139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:21.665071 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:21.665139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:21.665110 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:22.669641 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:22.669594 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" event={"ID":"494ff58d-8474-487f-8bbb-a7d068c56a65","Type":"ContainerStarted","Data":"51b56061d4a786e1e9611ba54f6b44d29c8895323564e242e27fa93a1aaddb87"} Oct 30 05:44:23.486800 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.486766 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:23.487004 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.486963 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:44:23.487084 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.487016 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 05:44:23.636486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.635383 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp"] Oct 30 05:44:23.641746 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.641717 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.654438 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.654407 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp"] Oct 30 05:44:23.660539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.660517 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e7a506-serving-cert\"" Oct 30 05:44:23.735797 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.735207 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68292eb0-96f8-4e6f-9668-3dd4b531496a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.735797 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.735296 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-metrics-certs\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.735797 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.735408 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9lp8\" (UniqueName: \"kubernetes.io/projected/68292eb0-96f8-4e6f-9668-3dd4b531496a-kube-api-access-h9lp8\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.735797 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.735451 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.857570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.856229 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9lp8\" (UniqueName: \"kubernetes.io/projected/68292eb0-96f8-4e6f-9668-3dd4b531496a-kube-api-access-h9lp8\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.857570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.856316 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.857570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.856491 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68292eb0-96f8-4e6f-9668-3dd4b531496a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.857570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.856560 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-metrics-certs\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.859436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.859190 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68292eb0-96f8-4e6f-9668-3dd4b531496a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.870911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.870859 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.873366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.873296 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-metrics-certs\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.881178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.881102 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9lp8\" (UniqueName: \"kubernetes.io/projected/68292eb0-96f8-4e6f-9668-3dd4b531496a-kube-api-access-h9lp8\") pod \"maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:23.963007 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:23.962968 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:24.297136 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:24.297108 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp"] Oct 30 05:44:24.298956 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:44:24.298921 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68292eb0_96f8_4e6f_9668_3dd4b531496a.slice/crio-544fe7fdd05a9e438098a8a41cd525794c59d5e3a1a9b481ab1b7b98d32e1167 WatchSource:0}: Error finding container 544fe7fdd05a9e438098a8a41cd525794c59d5e3a1a9b481ab1b7b98d32e1167: Status 404 returned error can't find the container with id 544fe7fdd05a9e438098a8a41cd525794c59d5e3a1a9b481ab1b7b98d32e1167 Oct 30 05:44:24.486877 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:24.486846 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:44:24.487052 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:24.486896 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 05:44:24.649961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:24.649931 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 05:44:24.688827 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:24.688798 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" event={"ID":"68292eb0-96f8-4e6f-9668-3dd4b531496a","Type":"ContainerStarted","Data":"544fe7fdd05a9e438098a8a41cd525794c59d5e3a1a9b481ab1b7b98d32e1167"} Oct 30 05:44:25.487281 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.487245 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:44:25.487859 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.487300 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 05:44:25.654988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.654960 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 05:44:25.696079 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.696047 5072 generic.go:356] "Generic (PLEG): container finished" podID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerID="8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8" exitCode=0 Oct 30 05:44:25.696240 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.696124 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" event={"ID":"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2","Type":"ContainerDied","Data":"8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8"} Oct 30 05:44:25.696568 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.696549 5072 scope.go:122] "RemoveContainer" containerID="8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8" Oct 30 05:44:25.697859 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.697834 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" event={"ID":"68292eb0-96f8-4e6f-9668-3dd4b531496a","Type":"ContainerStarted","Data":"22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86"} Oct 30 05:44:25.698018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.697988 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:25.698062 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.698019 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:25.698096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.698075 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:25.698129 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.698109 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:25.851163 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:25.851096 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" podStartSLOduration=2.5012571660000003 podStartE2EDuration="2.851075185s" totalImagesPullingTime="349.818019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:25.842602735 +0000 UTC m=+825.446292022" watchObservedRunningTime="2026-10-30 05:44:25.851075185 +0000 UTC m=+825.454764443" Oct 30 05:44:26.486702 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:26.486670 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:44:26.486874 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:26.486718 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 05:44:26.706096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:26.706063 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" event={"ID":"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2","Type":"ContainerStarted","Data":"cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f"} Oct 30 05:44:27.249485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:27.248866 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:27.249485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:27.248992 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:44:27.249485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:27.249036 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 05:44:27.487293 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:27.487245 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:44:27.487473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:27.487315 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 05:44:28.245086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:28.245045 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:44:28.245539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:28.245126 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 05:44:28.490795 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:28.490764 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:28.491239 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:28.491225 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:28.491859 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:28.491843 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:44:29.245703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:29.245657 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:44:29.246202 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:29.245728 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 05:44:30.245976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:30.245935 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:44:30.246353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:30.245998 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 05:44:30.721489 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:44:30.720396 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:44:31.255485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:31.251883 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:44:31.255485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:31.251942 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 05:44:31.740484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:31.733635 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 05:44:32.509722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:32.509691 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:32.511534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:32.511512 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:32.521525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:32.521501 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 05:44:41.841909 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:41.841882 5072 generic.go:356] "Generic (PLEG): container finished" podID="fc8c667c-d266-42c4-a528-c8957316da9c" containerID="dfce374ad48b8e35a7e37e6b884c597910886ede3a60d79bebcfd414a61707be" exitCode=0 Oct 30 05:44:41.842280 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:41.841970 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" event={"ID":"fc8c667c-d266-42c4-a528-c8957316da9c","Type":"ContainerDied","Data":"dfce374ad48b8e35a7e37e6b884c597910886ede3a60d79bebcfd414a61707be"} Oct 30 05:44:41.842486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:41.842447 5072 scope.go:122] "RemoveContainer" containerID="dfce374ad48b8e35a7e37e6b884c597910886ede3a60d79bebcfd414a61707be" Oct 30 05:44:41.843884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:41.843857 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:41.843954 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:41.843925 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:41.843988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:41.843954 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:44:42.853329 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:42.853295 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" event={"ID":"fc8c667c-d266-42c4-a528-c8957316da9c","Type":"ContainerStarted","Data":"43437b170d9900a2d5f67edd8a8ea89bdaed43cddf1a7a73f4a2e2820970c3bb"} Oct 30 05:44:43.034189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:43.033882 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:43.034189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:43.034022 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:44:43.034189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:43.034081 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 05:44:44.031485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:44.030878 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:44:44.031485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:44.030947 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 05:44:45.032484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:45.031898 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:44:45.032484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:45.031966 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 05:44:46.030884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:46.030844 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:44:46.031164 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:46.031133 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 05:44:47.030969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:47.030931 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:44:47.031633 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:47.031580 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 05:44:48.347168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:48.347136 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:48.347686 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:48.347504 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:44:48.348903 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:44:48.348885 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 05:45:00.710284 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:45:00.710201 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:45:01.052988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.052910 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829"] Oct 30 05:45:01.056991 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.056972 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.071866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.071844 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-64123f-openshift-default-dockercfg-msqjh\"" Oct 30 05:45:01.086895 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.086858 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829"] Oct 30 05:45:01.209157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.209117 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.209332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.209182 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.209373 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.209322 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.209373 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.209357 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.209438 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.209380 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.209508 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.209493 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.209546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.209536 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfwvc\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-kube-api-access-gfwvc\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.209625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.209614 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.209659 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.209645 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.310602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.310510 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.310602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.310555 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.310602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.310573 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.310873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.310691 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfwvc\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-kube-api-access-gfwvc\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.310873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.310818 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.310873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.310858 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.311124 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.310890 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.311124 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.310935 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.311124 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.310986 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.311124 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.311068 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.311124 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.311090 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.311383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.311240 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.311383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.311302 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.311664 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.311641 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.313553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.313529 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.313683 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.313609 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.342327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.342290 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.342522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.342368 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfwvc\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-kube-api-access-gfwvc\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.376773 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.376738 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:01.745688 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.744968 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829"] Oct 30 05:45:01.747515 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:45:01.747452 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb62c6d0_9478_4a3e_b43b_9731333b9db4.slice/crio-213e0726cfad2f8bffc012a1d0bfcc126d34a241637a0346ed830845cff730ca WatchSource:0}: Error finding container 213e0726cfad2f8bffc012a1d0bfcc126d34a241637a0346ed830845cff730ca: Status 404 returned error can't find the container with id 213e0726cfad2f8bffc012a1d0bfcc126d34a241637a0346ed830845cff730ca Oct 30 05:45:01.750151 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.750101 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:45:01.750243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.750181 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:45:01.750243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.750211 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:45:01.989503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.988802 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" event={"ID":"cb62c6d0-9478-4a3e-b43b-9731333b9db4","Type":"ContainerStarted","Data":"fa9c28a6e7c4d14a8e45a4886496746dd1417e44b8a486b078ee9b88e49f4573"} Oct 30 05:45:01.989503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:01.988870 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" event={"ID":"cb62c6d0-9478-4a3e-b43b-9731333b9db4","Type":"ContainerStarted","Data":"213e0726cfad2f8bffc012a1d0bfcc126d34a241637a0346ed830845cff730ca"} Oct 30 05:45:02.036212 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:02.036087 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" podStartSLOduration=1.036066337 podStartE2EDuration="1.036066337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:02.034846787 +0000 UTC m=+861.638536068" watchObservedRunningTime="2026-10-30 05:45:02.036066337 +0000 UTC m=+861.639755595" Oct 30 05:45:02.377126 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:02.377083 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:02.378818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:02.378792 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:45:02.378951 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:02.378839 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 05:45:03.378012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:03.377972 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:45:03.378511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:03.378040 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 05:45:04.377803 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:04.377755 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:45:04.378027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:04.377828 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 05:45:05.377917 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:05.377882 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:45:05.378103 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:05.377952 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 05:45:06.377757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:06.377717 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:45:06.377946 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:06.377784 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 05:45:07.377852 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:07.377807 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:45:07.378279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:07.377877 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 05:45:08.384901 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:08.384863 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:08.385384 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:08.385232 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:08.386174 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:08.386151 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:45:30.711520 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:45:30.711483 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:45:40.611246 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:40.611211 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:45:40.613964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:45:40.613939 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:46:00.714339 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:46:00.714297 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:46:02.003540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.003504 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42"] Oct 30 05:46:02.007447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.007431 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.024289 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.024257 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-be78f2-openshift-default-dockercfg-8twx9\"" Oct 30 05:46:02.039078 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.039035 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42"] Oct 30 05:46:02.115100 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.115066 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.115303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.115119 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.115303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.115153 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.115397 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.115330 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.115397 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.115369 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.115525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.115484 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.115580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.115544 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2fhl\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-kube-api-access-q2fhl\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.115663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.115637 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.115770 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.115751 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82915e36-0e03-48c5-8d83-942037d0e5f5-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.216874 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.216840 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82915e36-0e03-48c5-8d83-942037d0e5f5-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217084 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.216901 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217084 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.216934 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217084 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.216955 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217084 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217002 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217108 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217177 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217228 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2fhl\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-kube-api-access-q2fhl\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217279 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217443 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217494 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217713 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217738 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82915e36-0e03-48c5-8d83-942037d0e5f5-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.217966 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.217893 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.219878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.219851 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.220014 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.219995 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.245661 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.245627 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.246719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.246697 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2fhl\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-kube-api-access-q2fhl\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.320940 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.320856 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:02.641374 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.641346 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42"] Oct 30 05:46:02.644232 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:46:02.644204 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82915e36_0e03_48c5_8d83_942037d0e5f5.slice/crio-228fe673fda5e87b699d520038dc8e2b2dbbf3d6a3c32daf0a5609b141728586 WatchSource:0}: Error finding container 228fe673fda5e87b699d520038dc8e2b2dbbf3d6a3c32daf0a5609b141728586: Status 404 returned error can't find the container with id 228fe673fda5e87b699d520038dc8e2b2dbbf3d6a3c32daf0a5609b141728586 Oct 30 05:46:02.646497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.646422 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:46:02.646696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.646668 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:46:02.646783 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:02.646719 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:46:03.406087 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:03.406052 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" event={"ID":"82915e36-0e03-48c5-8d83-942037d0e5f5","Type":"ContainerStarted","Data":"6d84558119fb5bc081798d9d481fc59cdf7ec223afe177983a954e7873a58de0"} Oct 30 05:46:03.406603 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:03.406125 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" event={"ID":"82915e36-0e03-48c5-8d83-942037d0e5f5","Type":"ContainerStarted","Data":"228fe673fda5e87b699d520038dc8e2b2dbbf3d6a3c32daf0a5609b141728586"} Oct 30 05:46:03.482187 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:03.482138 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" podStartSLOduration=2.482121626 podStartE2EDuration="2.482121626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:03.469857899 +0000 UTC m=+923.073547182" watchObservedRunningTime="2026-10-30 05:46:03.482121626 +0000 UTC m=+923.085810884" Oct 30 05:46:04.333490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:04.329546 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:04.333490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:04.329735 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:46:04.333490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:04.329782 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 05:46:05.324875 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:05.324827 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:46:05.325505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:05.325471 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 05:46:06.326486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:06.322811 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:46:06.326486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:06.322874 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 05:46:07.327594 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:07.327557 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:46:07.328193 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:07.328162 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 05:46:08.330485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:08.321796 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:46:08.330485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:08.321854 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 05:46:09.587098 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:09.587061 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:09.588147 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:09.588122 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:09.589747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:09.589726 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:46:16.495679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.495643 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t"] Oct 30 05:46:16.501324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.501303 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.540849 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.540816 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t"] Oct 30 05:46:16.561952 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.561912 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.562126 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.561991 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.562172 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.562116 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.562286 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.562267 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e19c9f-4d40-499d-bf2e-4c92e850391b-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.562340 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.562308 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.562399 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.562348 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ckj\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-kube-api-access-c5ckj\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.562501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.562443 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.562579 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.562554 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.562645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.562626 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664206 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664170 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e19c9f-4d40-499d-bf2e-4c92e850391b-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664206 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664211 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664233 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ckj\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-kube-api-access-c5ckj\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664264 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664298 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664353 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664396 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664490 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664536 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.664765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.664709 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.665232 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.665139 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e19c9f-4d40-499d-bf2e-4c92e850391b-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.666529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.666439 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.666529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.666498 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.666701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.666546 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.667278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.667252 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.667780 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.667757 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.691844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.691782 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.691974 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.691913 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ckj\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-kube-api-access-c5ckj\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.821751 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.821668 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm"] Oct 30 05:46:16.827024 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.826999 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:16.831315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.831293 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:16.842602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.842578 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fffaa8-openshift-default-dockercfg-fhmlj\"" Oct 30 05:46:16.856429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.856392 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm"] Oct 30 05:46:16.971219 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.971165 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:16.971387 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.971231 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:16.971452 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.971302 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:16.971452 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.971432 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:16.971606 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.971496 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:16.971606 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.971565 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:16.971606 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.971598 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8e9c3-0956-46e3-bcb7-02225a62c460-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:16.971763 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.971664 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2l9p\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-kube-api-access-q2l9p\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:16.971763 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:16.971728 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073358 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073402 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073421 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073511 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073546 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8e9c3-0956-46e3-bcb7-02225a62c460-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073568 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2l9p\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-kube-api-access-q2l9p\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073643 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073923 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073785 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073923 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073815 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073923 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073828 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073923 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073862 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.073923 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.073909 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.074189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.074169 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.074282 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.074263 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8e9c3-0956-46e3-bcb7-02225a62c460-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.076047 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.076025 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.076159 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.076067 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.099361 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.099332 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2l9p\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-kube-api-access-q2l9p\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.100313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.100295 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.145511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.145478 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:17.206583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.206559 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t"] Oct 30 05:46:17.218703 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:46:17.218665 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e19c9f_4d40_499d_bf2e_4c92e850391b.slice/crio-3810cf1e78110e503106a09dbb91199d20dcdc275cdc7ac1cb12ab47bb255f21 WatchSource:0}: Error finding container 3810cf1e78110e503106a09dbb91199d20dcdc275cdc7ac1cb12ab47bb255f21: Status 404 returned error can't find the container with id 3810cf1e78110e503106a09dbb91199d20dcdc275cdc7ac1cb12ab47bb255f21 Oct 30 05:46:17.222871 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.221880 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:46:17.222871 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.221971 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:46:17.222871 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.222014 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:46:17.458273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.458237 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm"] Oct 30 05:46:17.458717 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:46:17.458688 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b8e9c3_0956_46e3_bcb7_02225a62c460.slice/crio-ebd7c809af2643c36ead0ad1d16acf81caee58bd6e42f4cc031f95e687fc0b3a WatchSource:0}: Error finding container ebd7c809af2643c36ead0ad1d16acf81caee58bd6e42f4cc031f95e687fc0b3a: Status 404 returned error can't find the container with id ebd7c809af2643c36ead0ad1d16acf81caee58bd6e42f4cc031f95e687fc0b3a Oct 30 05:46:17.461546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.461514 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:46:17.461627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.461578 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:46:17.461627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.461606 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:46:17.570485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.566426 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" event={"ID":"76b8e9c3-0956-46e3-bcb7-02225a62c460","Type":"ContainerStarted","Data":"66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909"} Oct 30 05:46:17.570485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.566525 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" event={"ID":"76b8e9c3-0956-46e3-bcb7-02225a62c460","Type":"ContainerStarted","Data":"ebd7c809af2643c36ead0ad1d16acf81caee58bd6e42f4cc031f95e687fc0b3a"} Oct 30 05:46:17.570485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.569190 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" event={"ID":"80e19c9f-4d40-499d-bf2e-4c92e850391b","Type":"ContainerStarted","Data":"e6a5ae7b68936593dbfddbddea1f8b06c10b4ec5d8c7608a532f7beeec4afe1d"} Oct 30 05:46:17.570485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.569238 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" event={"ID":"80e19c9f-4d40-499d-bf2e-4c92e850391b","Type":"ContainerStarted","Data":"3810cf1e78110e503106a09dbb91199d20dcdc275cdc7ac1cb12ab47bb255f21"} Oct 30 05:46:17.621358 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.621290 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" podStartSLOduration=1.621269748 podStartE2EDuration="1.621269748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:17.617492505 +0000 UTC m=+937.221181763" watchObservedRunningTime="2026-10-30 05:46:17.621269748 +0000 UTC m=+937.224959008" Oct 30 05:46:17.667893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.666125 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" podStartSLOduration=1.6661025839999999 podStartE2EDuration="1.666102584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:17.66237652 +0000 UTC m=+937.266065781" watchObservedRunningTime="2026-10-30 05:46:17.666102584 +0000 UTC m=+937.269791842" Oct 30 05:46:17.841542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:17.840562 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:18.152485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:18.152138 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:18.152485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:18.152321 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:46:18.152485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:18.152367 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 05:46:18.837484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:18.837333 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 05:46:18.837484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:18.837412 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 05:46:19.155486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:19.151992 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:46:19.155486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:19.152053 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 05:46:19.836484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:19.835863 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 05:46:19.836484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:19.835928 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 05:46:20.154484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:20.153807 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:46:20.154484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:20.153864 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 05:46:20.840484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:20.833993 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 05:46:20.840484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:20.834114 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 05:46:21.150482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:21.149875 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:46:21.150482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:21.149941 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 05:46:21.832894 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:21.832854 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 05:46:21.833548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:21.833514 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 05:46:22.149866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:22.149827 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:46:22.150165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:22.150132 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 05:46:22.839516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:22.839230 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 05:46:22.839516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:22.839324 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 05:46:23.153267 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:23.153232 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:46:23.153573 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:23.153544 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 05:46:24.105149 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:24.105111 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:24.105921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:24.105898 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:24.107778 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:24.107758 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 05:46:24.416494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:24.416373 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:24.417233 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:24.417204 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:24.418176 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:46:24.418155 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 05:46:30.710686 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:46:30.710635 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:47:00.714591 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:47:00.714547 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:47:02.227291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.227259 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5"] Oct 30 05:47:02.231652 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.231625 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.255254 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.255229 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5"] Oct 30 05:47:02.300060 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.300022 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.300252 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.300078 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.300252 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.300216 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.300391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.300272 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.300391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.300351 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.300526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.300415 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.300584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.300529 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.300632 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.300591 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2bp\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-kube-api-access-wh2bp\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.300688 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.300663 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402070 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402035 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402257 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402111 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402257 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402147 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402422 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402651 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402624 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402656 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402702 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2bp\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-kube-api-access-wh2bp\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402703 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402762 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402999 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402814 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.402999 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.402844 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.403145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.403121 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.403272 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.403252 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.403370 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.403350 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.407122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.407075 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.410201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.410180 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.425928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.425898 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2bp\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-kube-api-access-wh2bp\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.426119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.426088 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.546114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.546012 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:02.868539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.868485 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5"] Oct 30 05:47:02.869961 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:47:02.869930 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7caf44c_c7f8_4daa_8957_1ea7b2729676.slice/crio-900ce09f4e4ea3e9b2e60c3fdbb67233af3d91189adfb4724025ad7df110cdb1 WatchSource:0}: Error finding container 900ce09f4e4ea3e9b2e60c3fdbb67233af3d91189adfb4724025ad7df110cdb1: Status 404 returned error can't find the container with id 900ce09f4e4ea3e9b2e60c3fdbb67233af3d91189adfb4724025ad7df110cdb1 Oct 30 05:47:02.872371 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.872341 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:02.872471 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.872403 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:02.872471 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:02.872429 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:03.193671 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.193586 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" event={"ID":"e7caf44c-c7f8-4daa-8957-1ea7b2729676","Type":"ContainerStarted","Data":"bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b"} Oct 30 05:47:03.193671 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.193643 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" event={"ID":"e7caf44c-c7f8-4daa-8957-1ea7b2729676","Type":"ContainerStarted","Data":"900ce09f4e4ea3e9b2e60c3fdbb67233af3d91189adfb4724025ad7df110cdb1"} Oct 30 05:47:03.241244 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.241189 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" podStartSLOduration=1.241175751 podStartE2EDuration="1.241175751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:03.238330256 +0000 UTC m=+982.842019513" watchObservedRunningTime="2026-10-30 05:47:03.241175751 +0000 UTC m=+982.844865065" Oct 30 05:47:03.547135 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.547035 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:03.548823 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.548795 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:03.548959 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.548845 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 05:47:03.735680 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.732054 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm"] Oct 30 05:47:03.759063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.759033 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm"] Oct 30 05:47:03.759222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.759174 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.851536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.846102 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2w9\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-kube-api-access-fl2w9\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.851536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.846165 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.851536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.846225 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.851536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.846291 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.851536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.846355 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.851536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.846414 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.851536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.846552 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.851536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.846613 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.851536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.846645 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.949579 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.949650 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.949680 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.949705 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.949756 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2w9\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-kube-api-access-fl2w9\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.949784 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.949818 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.949859 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.949891 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.950538 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.950803 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.950961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.950853 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.951505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.951026 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.951505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.951330 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.953163 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.953122 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.953815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.953793 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.973897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.973868 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:03.973998 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:03.973921 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2w9\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-kube-api-access-fl2w9\") pod \"maas-default-gateway-openshift-default-84587756dc-5qfjm\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:04.076020 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:04.075980 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:04.441916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:04.441885 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm"] Oct 30 05:47:04.445594 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:47:04.445551 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aab9b7f_029d_4619_9cbc_0f59084d6d20.slice/crio-537848c91ca92fe3c65ba3a327227ebd905ee5749b45f48f51e65f60e0581936 WatchSource:0}: Error finding container 537848c91ca92fe3c65ba3a327227ebd905ee5749b45f48f51e65f60e0581936: Status 404 returned error can't find the container with id 537848c91ca92fe3c65ba3a327227ebd905ee5749b45f48f51e65f60e0581936 Oct 30 05:47:04.448095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:04.448052 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:04.448196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:04.448145 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:04.448196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:04.448187 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:04.547154 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:04.547120 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:04.547330 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:04.547188 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 05:47:05.231314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:05.231280 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" event={"ID":"1aab9b7f-029d-4619-9cbc-0f59084d6d20","Type":"ContainerStarted","Data":"1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a"} Oct 30 05:47:05.231597 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:05.231571 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" event={"ID":"1aab9b7f-029d-4619-9cbc-0f59084d6d20","Type":"ContainerStarted","Data":"537848c91ca92fe3c65ba3a327227ebd905ee5749b45f48f51e65f60e0581936"} Oct 30 05:47:05.290808 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:05.290757 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" podStartSLOduration=2.290733485 podStartE2EDuration="2.290733485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:05.290040162 +0000 UTC m=+984.893729420" watchObservedRunningTime="2026-10-30 05:47:05.290733485 +0000 UTC m=+984.894422742" Oct 30 05:47:05.550538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:05.546872 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:05.550538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:05.546957 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 05:47:06.076407 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:06.076374 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:06.077949 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:06.077911 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5qfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:47:06.078015 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:06.077977 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 05:47:06.555563 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:06.555529 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:06.557268 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:06.557237 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 05:47:07.077288 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:07.077238 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5qfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:47:07.077506 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:07.077303 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 05:47:07.555005 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:07.554965 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:07.555314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:07.555280 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 05:47:08.076734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:08.076685 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5qfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:47:08.077239 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:08.076749 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 05:47:08.551865 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:08.551826 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:08.552615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:08.552575 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 05:47:09.076952 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:09.076921 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5qfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:47:09.077356 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:09.076973 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 05:47:09.860801 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:09.860770 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:09.861928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:09.861908 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:09.863177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:09.863162 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:47:10.077242 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:10.077207 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5qfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:47:10.077728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:10.077276 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 05:47:11.077251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:11.077219 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5qfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:47:11.077780 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:11.077281 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 05:47:12.077175 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:12.077132 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-5qfjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:47:12.077356 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:12.077188 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 05:47:13.006873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.001994 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx"] Oct 30 05:47:13.064808 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.064767 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.082196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.082158 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx"] Oct 30 05:47:13.092199 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.092182 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"default-dockercfg-bv9bw\"" Oct 30 05:47:13.092501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.092486 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"openshift-service-ca.crt\"" Oct 30 05:47:13.092630 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.092578 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"kube-root-ca.crt\"" Oct 30 05:47:13.182263 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.182230 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"distinct-2-w4-698ab2-kserve-self-signed-certs\"" Oct 30 05:47:13.228947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.227211 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-home\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.228947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.227361 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tls-certs\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.228947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.227395 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-model-cache\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.228947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.227476 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-dshm\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.228947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.227547 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt5pz\" (UniqueName: \"kubernetes.io/projected/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-kube-api-access-lt5pz\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.228947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.227601 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tmp-dir\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.343543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.341236 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-dshm\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.343543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.341431 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt5pz\" (UniqueName: \"kubernetes.io/projected/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-kube-api-access-lt5pz\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.343543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.341546 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tmp-dir\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.343543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.341645 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-home\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.343543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.341774 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tls-certs\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.343543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.341808 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-model-cache\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.361116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.361085 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tmp-dir\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.378292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.377847 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-model-cache\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.378816 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.378787 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-home\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.396772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.396697 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt5pz\" (UniqueName: \"kubernetes.io/projected/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-kube-api-access-lt5pz\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.396957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.396894 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-dshm\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.400489 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.400448 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tls-certs\") pod \"distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:13.510183 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.510149 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:13.511825 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.511803 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:13.515582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.515558 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:47:13.693921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:13.692859 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:14.306474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:14.306423 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx"] Oct 30 05:47:14.307898 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:47:14.307860 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4285a424_8b5e_4aa1_b77e_3ad6fe415b62.slice/crio-731a249418a8f3bde4e9bbbfcd419f2b629b1c4b1dab1076e9b8d8c6eebccc47 WatchSource:0}: Error finding container 731a249418a8f3bde4e9bbbfcd419f2b629b1c4b1dab1076e9b8d8c6eebccc47: Status 404 returned error can't find the container with id 731a249418a8f3bde4e9bbbfcd419f2b629b1c4b1dab1076e9b8d8c6eebccc47 Oct 30 05:47:14.386193 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:14.385262 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" event={"ID":"4285a424-8b5e-4aa1-b77e-3ad6fe415b62","Type":"ContainerStarted","Data":"731a249418a8f3bde4e9bbbfcd419f2b629b1c4b1dab1076e9b8d8c6eebccc47"} Oct 30 05:47:15.391532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:15.391498 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" event={"ID":"4285a424-8b5e-4aa1-b77e-3ad6fe415b62","Type":"ContainerStarted","Data":"cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6"} Oct 30 05:47:15.434092 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:15.434035 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podStartSLOduration=3.434020355 podStartE2EDuration="3.434020355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:15.431702985 +0000 UTC m=+995.035392242" watchObservedRunningTime="2026-10-30 05:47:15.434020355 +0000 UTC m=+995.037709612" Oct 30 05:47:15.694868 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:15.694794 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:16.802969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.802936 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4"] Oct 30 05:47:16.808756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.808733 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:16.841207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.841179 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4"] Oct 30 05:47:16.915445 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.915406 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:16.915649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.915506 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:16.915649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.915564 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:16.915649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.915608 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5xtn\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-kube-api-access-g5xtn\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:16.915770 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.915704 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:16.915770 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.915731 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:16.915867 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.915847 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:16.915929 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.915918 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:16.916007 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:16.915995 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.017606 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.017575 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.017798 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.017666 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.017798 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.017710 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.017798 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.017733 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.017798 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.017771 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5xtn\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-kube-api-access-g5xtn\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.018021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.017833 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.018021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.017861 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.018021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.017940 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.018172 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.018035 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.018676 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.018637 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.018829 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.018744 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.018931 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.018909 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.019083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.019056 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.019614 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.019586 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.021046 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.021015 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.021427 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.021407 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.082084 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.082016 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.087295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.087268 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5xtn\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-kube-api-access-g5xtn\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.123015 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.122982 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:17.445056 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:47:17.445033 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac75576_ed42_464e_b4ce_b7659ff9c7e1.slice/crio-c8bd1a1c2bb058a777e712a62188606b85bd3e5c0b5ff46b0ec33383968de767 WatchSource:0}: Error finding container c8bd1a1c2bb058a777e712a62188606b85bd3e5c0b5ff46b0ec33383968de767: Status 404 returned error can't find the container with id c8bd1a1c2bb058a777e712a62188606b85bd3e5c0b5ff46b0ec33383968de767 Oct 30 05:47:17.445565 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.445541 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4"] Oct 30 05:47:17.447628 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.447597 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:17.447709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.447659 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:17.447709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.447691 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:17.767293 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.767208 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5"] Oct 30 05:47:17.772706 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.772510 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.789393 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.789339 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"kube-root-ca.crt\"" Oct 30 05:47:17.801091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.801058 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"default-dockercfg-279cs\"" Oct 30 05:47:17.801091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.801084 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"distinct-w3-fffaa8-kserve-self-signed-certs\"" Oct 30 05:47:17.801313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.801138 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"openshift-service-ca.crt\"" Oct 30 05:47:17.804259 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.804220 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5"] Oct 30 05:47:17.828227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.827804 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-home\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.828227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.827851 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-dshm\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.828227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.827920 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tgb\" (UniqueName: \"kubernetes.io/projected/d7452308-ea76-448e-a885-300436798e92-kube-api-access-r5tgb\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.828227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.827960 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-model-cache\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.828227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.828007 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-tmp-dir\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.828227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.828053 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7452308-ea76-448e-a885-300436798e92-tls-certs\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.929775 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.929739 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-home\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.929775 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.929776 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-dshm\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.930013 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.929831 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tgb\" (UniqueName: \"kubernetes.io/projected/d7452308-ea76-448e-a885-300436798e92-kube-api-access-r5tgb\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.930013 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.929858 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-model-cache\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.930013 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.929892 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-tmp-dir\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.930258 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.930232 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-home\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.930336 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.930281 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-model-cache\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.930523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.930495 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-tmp-dir\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.930830 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.930807 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7452308-ea76-448e-a885-300436798e92-tls-certs\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.932543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.932517 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-dshm\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.933487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.933444 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7452308-ea76-448e-a885-300436798e92-tls-certs\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:17.986616 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:17.986581 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tgb\" (UniqueName: \"kubernetes.io/projected/d7452308-ea76-448e-a885-300436798e92-kube-api-access-r5tgb\") pod \"distinct-w3-fffaa8-kserve-74d88cf5f8-566j5\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:18.110483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:18.104240 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:18.437838 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:18.437678 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" event={"ID":"3ac75576-ed42-464e-b4ce-b7659ff9c7e1","Type":"ContainerStarted","Data":"cc62c0a1387d6661e48e98e65068c998b46414481054fee719ce6a05051a94dd"} Oct 30 05:47:18.439119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:18.438061 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" event={"ID":"3ac75576-ed42-464e-b4ce-b7659ff9c7e1","Type":"ContainerStarted","Data":"c8bd1a1c2bb058a777e712a62188606b85bd3e5c0b5ff46b0ec33383968de767"} Oct 30 05:47:18.480925 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:18.480902 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5"] Oct 30 05:47:18.482641 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:47:18.482612 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7452308_ea76_448e_a885_300436798e92.slice/crio-69cd1e45af2d5e31479b9304c66053943ccf417cb259c9d5851bec0a01fb3343 WatchSource:0}: Error finding container 69cd1e45af2d5e31479b9304c66053943ccf417cb259c9d5851bec0a01fb3343: Status 404 returned error can't find the container with id 69cd1e45af2d5e31479b9304c66053943ccf417cb259c9d5851bec0a01fb3343 Oct 30 05:47:18.492731 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:18.492692 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" podStartSLOduration=2.492678972 podStartE2EDuration="2.492678972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:18.483530281 +0000 UTC m=+998.087219567" watchObservedRunningTime="2026-10-30 05:47:18.492678972 +0000 UTC m=+998.096368326" Oct 30 05:47:19.129485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:19.128537 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:19.129485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:19.128699 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:19.129485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:19.128744 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:47:19.446355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:19.446258 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" event={"ID":"d7452308-ea76-448e-a885-300436798e92","Type":"ContainerStarted","Data":"d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8"} Oct 30 05:47:19.446355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:19.446309 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" event={"ID":"d7452308-ea76-448e-a885-300436798e92","Type":"ContainerStarted","Data":"69cd1e45af2d5e31479b9304c66053943ccf417cb259c9d5851bec0a01fb3343"} Oct 30 05:47:19.487578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:19.487517 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podStartSLOduration=2.4874978 podStartE2EDuration="2.4874978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:19.486497427 +0000 UTC m=+999.090186705" watchObservedRunningTime="2026-10-30 05:47:19.4874978 +0000 UTC m=+999.091187059" Oct 30 05:47:20.105970 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:20.105928 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:20.126879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:20.126844 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:20.127528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:20.127123 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:47:21.130483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:21.129856 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:21.130483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:21.129929 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:47:22.125890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:22.125852 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:22.126173 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:22.126141 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:47:23.126481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:23.125854 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:23.126481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:23.125916 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:47:23.695067 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:23.695028 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:23.708016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:23.707990 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:24.123548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:24.123509 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:24.123731 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:24.123562 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 05:47:24.491123 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:24.491041 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:47:25.128777 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:25.128746 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:25.129615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:25.129584 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:25.130421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:25.130402 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:47:28.112564 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:28.112529 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:28.193497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:28.192098 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:28.635279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:28.635247 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:47:30.712557 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:47:30.712519 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:47:31.736096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.736058 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq"] Oct 30 05:47:31.747043 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.746777 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.767544 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.767507 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq"] Oct 30 05:47:31.839888 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.839849 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.840074 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.839912 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.840127 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.840102 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.840278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.840251 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.840434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.840332 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.840434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.840370 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtp2\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-kube-api-access-5xtp2\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.840434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.840413 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.840434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.840433 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9186dda-184a-47f6-913e-e3ba748da94d-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.840707 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.840524 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9186dda-184a-47f6-913e-e3ba748da94d-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.950542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.949589 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.950542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.949641 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtp2\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-kube-api-access-5xtp2\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.950542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.949702 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.950542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.949726 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9186dda-184a-47f6-913e-e3ba748da94d-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.950542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.949754 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9186dda-184a-47f6-913e-e3ba748da94d-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.950542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.949843 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.950542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.949887 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.950542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.949972 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.950542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.950096 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.951145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.950563 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.951145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.950800 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.968813 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.968105 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.968813 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.968486 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.969948 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.969910 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9186dda-184a-47f6-913e-e3ba748da94d-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.971805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.971779 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9186dda-184a-47f6-913e-e3ba748da94d-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.975472 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.975431 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtp2\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-kube-api-access-5xtp2\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.996377 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.996036 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:31.999229 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:31.999200 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:32.065277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:32.065239 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:32.386735 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:32.386711 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq"] Oct 30 05:47:32.388415 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:47:32.388370 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9186dda_184a_47f6_913e_e3ba748da94d.slice/crio-dcb8cdc55ca845aabb2511371efb23b805aee59b5adb46e6826dc4fa42c5384b WatchSource:0}: Error finding container dcb8cdc55ca845aabb2511371efb23b805aee59b5adb46e6826dc4fa42c5384b: Status 404 returned error can't find the container with id dcb8cdc55ca845aabb2511371efb23b805aee59b5adb46e6826dc4fa42c5384b Oct 30 05:47:32.391190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:32.391158 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:32.391291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:32.391232 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:32.391291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:32.391261 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:32.649709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:32.649609 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" event={"ID":"c9186dda-184a-47f6-913e-e3ba748da94d","Type":"ContainerStarted","Data":"ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686"} Oct 30 05:47:32.649709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:32.649671 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" event={"ID":"c9186dda-184a-47f6-913e-e3ba748da94d","Type":"ContainerStarted","Data":"dcb8cdc55ca845aabb2511371efb23b805aee59b5adb46e6826dc4fa42c5384b"} Oct 30 05:47:32.696199 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:32.696136 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" podStartSLOduration=1.6961167270000002 podStartE2EDuration="1.696116727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:32.694524311 +0000 UTC m=+1012.298213593" watchObservedRunningTime="2026-10-30 05:47:32.696116727 +0000 UTC m=+1012.299805985" Oct 30 05:47:33.066342 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:33.066259 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:33.068197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:33.068170 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:47:33.068317 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:33.068231 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:47:34.066168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:34.066127 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:47:34.066354 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:34.066191 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:47:35.066051 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:35.066016 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:47:35.066230 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:35.066071 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:47:36.066037 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:36.065998 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:47:36.066578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:36.066067 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:47:37.065900 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:37.065865 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:47:37.066076 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:37.065917 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:47:38.066149 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:38.066108 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:47:38.066572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:38.066173 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:47:39.068208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:39.068171 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 05:47:39.068844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:39.068812 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 05:47:40.071692 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:40.071657 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:40.072145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:40.072127 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:40.072816 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:40.072799 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:47:46.596513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.596427 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6"] Oct 30 05:47:46.603066 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.603026 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.640580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.640543 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6"] Oct 30 05:47:46.721503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.719901 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfpj\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-kube-api-access-qrfpj\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.721503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.720112 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157a8419-22b9-4588-870a-0245bbefbf39-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.721503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.720215 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.721503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.720361 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.721503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.720447 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157a8419-22b9-4588-870a-0245bbefbf39-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.721503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.720527 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.721503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.720569 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.721503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.720602 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.721503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.720658 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.822127 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.822071 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157a8419-22b9-4588-870a-0245bbefbf39-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.822322 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.822141 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.822322 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.822196 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.822322 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.822235 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.822536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.822333 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.822536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.822417 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfpj\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-kube-api-access-qrfpj\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.822648 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.822585 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157a8419-22b9-4588-870a-0245bbefbf39-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.822705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.822639 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.822826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.822792 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.823309 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.823286 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.823937 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.823909 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.824044 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.824002 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.824298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.824265 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.825472 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.825427 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157a8419-22b9-4588-870a-0245bbefbf39-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.827412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.827388 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157a8419-22b9-4588-870a-0245bbefbf39-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.838778 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.838751 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.847623 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.847506 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfpj\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-kube-api-access-qrfpj\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.851404 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.851364 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.915036 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.913731 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx"] Oct 30 05:47:46.920745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.920543 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:46.921579 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:46.921210 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.045654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.044950 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.045654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.044999 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43331d15-6de3-4311-995b-30953745e472-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.045654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.045036 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.045654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.045087 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.045654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.045137 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.045654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.045269 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.045654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.045296 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43331d15-6de3-4311-995b-30953745e472-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.045654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.045338 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.045654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.045421 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48dxc\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-kube-api-access-48dxc\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.104010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.103965 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx"] Oct 30 05:47:47.154043 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.151591 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.154407 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.154378 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48dxc\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-kube-api-access-48dxc\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.154759 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.154735 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.157584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.157527 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43331d15-6de3-4311-995b-30953745e472-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.157765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.157742 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.157921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.157905 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.158691 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.158665 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.159053 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.159031 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.159192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.159175 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43331d15-6de3-4311-995b-30953745e472-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.162710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.161511 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.162861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.162153 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43331d15-6de3-4311-995b-30953745e472-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.162962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.162375 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.163392 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.163371 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.163837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.163816 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.257789 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.245943 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43331d15-6de3-4311-995b-30953745e472-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.257789 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.249904 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.258654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.258601 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.274133 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.274101 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48dxc\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-kube-api-access-48dxc\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:47.438032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:47.437703 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:48.107371 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.097138 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6"] Oct 30 05:47:48.115426 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.115365 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:48.115587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.115488 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:48.115587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.115541 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:48.420510 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.420477 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx"] Oct 30 05:47:48.428916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.428841 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:48.429159 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.429138 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:48.429297 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.429281 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:47:48.900158 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.900123 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" event={"ID":"157a8419-22b9-4588-870a-0245bbefbf39","Type":"ContainerStarted","Data":"60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1"} Oct 30 05:47:48.900365 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.900181 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" event={"ID":"157a8419-22b9-4588-870a-0245bbefbf39","Type":"ContainerStarted","Data":"a812f59676119497e254b54befc91fdf0312b4e52664399079cbf2a3314a6570"} Oct 30 05:47:48.902021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.901998 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" event={"ID":"43331d15-6de3-4311-995b-30953745e472","Type":"ContainerStarted","Data":"cef277d46ad45c013742e1c289c53a449969c62cda4cc483dc843bea73f45b20"} Oct 30 05:47:48.902140 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.902034 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" event={"ID":"43331d15-6de3-4311-995b-30953745e472","Type":"ContainerStarted","Data":"5b812d98dc8cb2fca39ac85361e0bd998924a5762cce38eec0e9f5b820b948a2"} Oct 30 05:47:48.921331 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.921284 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:48.970270 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:48.970221 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" podStartSLOduration=2.970206605 podStartE2EDuration="2.970206605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:48.958098192 +0000 UTC m=+1028.561787454" watchObservedRunningTime="2026-10-30 05:47:48.970206605 +0000 UTC m=+1028.573895858" Oct 30 05:47:49.016884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:49.008920 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" podStartSLOduration=3.008900828 podStartE2EDuration="3.008900828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:49.00228684 +0000 UTC m=+1028.605976090" watchObservedRunningTime="2026-10-30 05:47:49.008900828 +0000 UTC m=+1028.612590100" Oct 30 05:47:49.447523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:49.447444 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:49.455709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:49.455676 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:49.455938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:49.455913 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:47:49.936485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:49.931578 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:49.936485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:49.931676 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:47:50.448486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:50.441893 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:50.448486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:50.441952 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:47:50.929955 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:50.929918 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:50.933479 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:50.930199 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:47:51.441484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:51.440858 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:51.441484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:51.440921 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:47:51.931484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:51.927870 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:51.931484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:51.927929 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:47:52.449653 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:52.449612 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:52.449948 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:52.449919 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:47:52.945781 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:52.945741 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:52.947770 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:52.947737 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:47:53.484580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:53.483889 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:53.484580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:53.483949 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:47:53.925323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:53.925283 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:53.925629 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:53.925601 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:47:54.442485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:54.441834 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:54.442485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:54.441892 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:47:54.926363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:54.925888 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:54.926363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:54.925957 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 05:47:55.438880 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:55.438841 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:55.439027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:55.438912 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 05:47:56.118746 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:56.118714 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:56.120156 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:56.120127 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:56.121354 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:56.121336 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:47:56.651752 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:56.651723 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:56.652260 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:56.652233 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:47:56.653322 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:47:56.653292 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:48:00.709826 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:48:00.709792 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:48:22.508661 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:22.508600 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:48:22.692529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:22.668918 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:30.710228 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:48:30.710190 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:48:31.739547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.739510 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w"] Oct 30 05:48:31.744298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.744270 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.781865 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.781833 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w"] Oct 30 05:48:31.784740 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.784712 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl"] Oct 30 05:48:31.788993 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.788970 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.810481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.810435 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl"] Oct 30 05:48:31.859543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.859513 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.859710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.859571 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.859710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.859640 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.859710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.859673 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.859834 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.859726 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.859834 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.859777 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.859907 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.859878 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.859948 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.859910 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.859989 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.859971 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.860069 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.860053 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.860111 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.860085 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.860165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.860151 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.860204 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.860174 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.860242 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.860210 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81f2fe-42c9-48c2-a934-6d45733a369f-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.860279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.860266 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.860330 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.860313 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpvv\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-kube-api-access-8lpvv\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.860371 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.860339 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxf6q\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-kube-api-access-lxf6q\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.860408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.860395 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.961243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961207 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.961243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961243 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.961522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961287 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81f2fe-42c9-48c2-a934-6d45733a369f-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.961522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961322 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.961522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961366 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpvv\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-kube-api-access-8lpvv\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.961522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961396 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxf6q\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-kube-api-access-lxf6q\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.961522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961439 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.961836 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961587 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.961836 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961641 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.961836 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961680 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.961836 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961714 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.961836 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961765 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.962131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961842 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.962131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961867 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.962131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961909 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.962131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961921 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.962131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.961933 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.962131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962070 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.962131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962127 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.962566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962173 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.962566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962196 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81f2fe-42c9-48c2-a934-6d45733a369f-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.962566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962210 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.962566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962430 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.962566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962430 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.962566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962529 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.962566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962536 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.963022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.962576 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.963506 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.963138 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.965401 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.965374 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.965401 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.965394 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.965700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.965678 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.966099 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.966074 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.996302 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.996218 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:31.996302 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.996258 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpvv\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-kube-api-access-8lpvv\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:31.998588 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:31.998562 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:32.002390 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.002364 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxf6q\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-kube-api-access-lxf6q\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:32.060403 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.060088 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:32.064305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.064268 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm"] Oct 30 05:48:32.068840 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.068820 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.100386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.099977 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm"] Oct 30 05:48:32.113491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.113443 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:32.167341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.164672 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.167341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.164907 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.167341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.164987 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.167341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.165069 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.167341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.165109 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbdp\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-kube-api-access-pqbdp\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.167341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.165150 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.167341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.165195 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.167341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.165242 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.167341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.165296 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.266834 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.266746 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.266834 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.266801 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.267072 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.266851 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.267072 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.266992 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.267072 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.267052 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.267214 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.267132 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.267214 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.267172 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbdp\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-kube-api-access-pqbdp\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.267283 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.267210 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.267283 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.267229 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.267283 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.267252 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.267933 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.267870 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.268048 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.267952 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.268216 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.268196 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.268278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.268232 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.270165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.270140 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.271587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.271540 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.292798 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.292414 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbdp\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-kube-api-access-pqbdp\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.293201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.293171 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.334416 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.334383 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp"] Oct 30 05:48:32.341689 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.341663 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.361992 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.361963 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp"] Oct 30 05:48:32.400865 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.400441 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w"] Oct 30 05:48:32.401966 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:48:32.401929 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e81f2fe_42c9_48c2_a934_6d45733a369f.slice/crio-46ce22ff87c72ca1a49c22fc961c98e22dd4e349d8454e99dd0ee9722727529a WatchSource:0}: Error finding container 46ce22ff87c72ca1a49c22fc961c98e22dd4e349d8454e99dd0ee9722727529a: Status 404 returned error can't find the container with id 46ce22ff87c72ca1a49c22fc961c98e22dd4e349d8454e99dd0ee9722727529a Oct 30 05:48:32.405340 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.405307 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:32.405536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.405370 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:32.405536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.405397 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:32.422190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.422165 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:32.467602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.467571 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl"] Oct 30 05:48:32.470133 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.470106 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.470252 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.470146 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwt6\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-kube-api-access-qzwt6\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.470319 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.470267 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.470386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.470324 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012b25d5-4f64-439a-a6cd-6ed87e029374-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.470386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.470356 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.470519 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.470404 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.470576 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.470519 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.470628 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.470601 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.470807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.470677 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.474427 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:48:32.474391 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd3ba4a_8402_4f98_9d16_b51b61baf453.slice/crio-ecf5e00006a9636e9ce14e0030df1a56d54e32066893258e191e9d4668260515 WatchSource:0}: Error finding container ecf5e00006a9636e9ce14e0030df1a56d54e32066893258e191e9d4668260515: Status 404 returned error can't find the container with id ecf5e00006a9636e9ce14e0030df1a56d54e32066893258e191e9d4668260515 Oct 30 05:48:32.478107 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.477679 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:32.478107 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.477765 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:32.478107 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.477806 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.572942 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.573050 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.573124 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.573153 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwt6\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-kube-api-access-qzwt6\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.573278 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.573329 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012b25d5-4f64-439a-a6cd-6ed87e029374-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.573356 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.573402 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.573496 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.573999 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.574253 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.574820 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.579896 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012b25d5-4f64-439a-a6cd-6ed87e029374-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.582863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.580163 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.634749 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.634343 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwt6\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-kube-api-access-qzwt6\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.639169 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.637605 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.667831 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.667798 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.672795 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.672724 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:32.784294 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.784247 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" event={"ID":"1e81f2fe-42c9-48c2-a934-6d45733a369f","Type":"ContainerStarted","Data":"6fcc86c96af62b9470f5159873cf2c80e36a7471d40eb60b57ce8d00dd8ffc9f"} Oct 30 05:48:32.784801 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.784317 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" event={"ID":"1e81f2fe-42c9-48c2-a934-6d45733a369f","Type":"ContainerStarted","Data":"46ce22ff87c72ca1a49c22fc961c98e22dd4e349d8454e99dd0ee9722727529a"} Oct 30 05:48:32.789936 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.789730 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" event={"ID":"fdd3ba4a-8402-4f98-9d16-b51b61baf453","Type":"ContainerStarted","Data":"7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b"} Oct 30 05:48:32.790102 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.789960 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" event={"ID":"fdd3ba4a-8402-4f98-9d16-b51b61baf453","Type":"ContainerStarted","Data":"ecf5e00006a9636e9ce14e0030df1a56d54e32066893258e191e9d4668260515"} Oct 30 05:48:32.808805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.808778 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm"] Oct 30 05:48:32.814780 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.814737 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:32.814916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.814830 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:32.814916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.814876 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:32.843480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.843414 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podStartSLOduration=1.843398424 podStartE2EDuration="1.843398424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:32.836693767 +0000 UTC m=+1072.440383037" watchObservedRunningTime="2026-10-30 05:48:32.843398424 +0000 UTC m=+1072.447087735" Oct 30 05:48:32.957639 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:32.957604 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:33.061613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.061216 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:33.063081 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.063048 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:33.063214 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.063114 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:33.115605 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.115515 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:33.116241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.115905 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:33.116241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.115961 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:33.329012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.328897 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podStartSLOduration=2.328873903 podStartE2EDuration="2.328873903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:32.885782892 +0000 UTC m=+1072.489472150" watchObservedRunningTime="2026-10-30 05:48:33.328873903 +0000 UTC m=+1072.932563161" Oct 30 05:48:33.369110 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.369062 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp"] Oct 30 05:48:33.377116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.376009 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:33.377116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.376090 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:33.377116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.376131 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:33.874531 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.869505 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" event={"ID":"012b25d5-4f64-439a-a6cd-6ed87e029374","Type":"ContainerStarted","Data":"555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1"} Oct 30 05:48:33.874531 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.869620 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" event={"ID":"012b25d5-4f64-439a-a6cd-6ed87e029374","Type":"ContainerStarted","Data":"c1c6aba02c10b38ff5bf378e5b31ffbc7a8172c99f3eb39d69a7052d197933bc"} Oct 30 05:48:33.886446 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.886408 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" event={"ID":"f05e3600-3aa0-4c56-97ec-5fa829f18c7e","Type":"ContainerStarted","Data":"d9617f2eaf4cc820d53984109f8e0f359a67f39e3829fc6caa31230c8883209a"} Oct 30 05:48:33.887344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.886718 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" event={"ID":"f05e3600-3aa0-4c56-97ec-5fa829f18c7e","Type":"ContainerStarted","Data":"cf43aaf58719976c2420f296af3a2703ee6ff347d51dc5fe60159569b25d2665"} Oct 30 05:48:33.951497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:33.949168 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podStartSLOduration=1.949145591 podStartE2EDuration="1.949145591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:33.94028135 +0000 UTC m=+1073.543970607" watchObservedRunningTime="2026-10-30 05:48:33.949145591 +0000 UTC m=+1073.552834865" Oct 30 05:48:34.047098 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.046516 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:34.080484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.076305 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:34.080484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.076364 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:34.100559 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.083658 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:34.100559 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.083713 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:34.100559 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.084840 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podStartSLOduration=2.084821277 podStartE2EDuration="2.084821277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:34.04964256 +0000 UTC m=+1073.653331983" watchObservedRunningTime="2026-10-30 05:48:34.084821277 +0000 UTC m=+1073.688510535" Oct 30 05:48:34.114898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.114858 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:34.115182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.115148 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:34.437486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.435835 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:34.461838 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.461793 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:48:34.462103 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.462070 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:48:34.991582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.989946 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:34.991582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:34.990003 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:35.074036 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:35.074003 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:35.074320 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:35.074285 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:35.127012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:35.126859 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:35.127012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:35.126926 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:35.427186 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:35.427150 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:48:35.427499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:35.427473 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:48:35.960887 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:35.960851 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:35.961182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:35.961154 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:36.064257 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:36.064222 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:36.071948 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:36.071909 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:36.242552 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:36.238112 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:36.242552 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:36.238181 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:36.466797 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:36.466764 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:48:36.467080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:36.467053 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:48:37.006406 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:37.005978 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:37.006406 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:37.006038 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:37.422513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:37.377721 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:37.422513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:37.377837 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:37.454360 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:37.429926 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:48:37.454360 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:37.429987 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:48:37.567879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:37.567841 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:37.568208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:37.568159 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:38.038487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:38.008800 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:38.051483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:38.038797 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:38.083688 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:38.083644 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:38.109596 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:38.109552 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:38.137567 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:38.137533 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:38.156480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:38.155944 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:38.475533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:38.465974 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:48:38.475533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:38.466043 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:48:38.984396 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:38.984361 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:39.000389 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:39.000348 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:39.085448 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:39.085411 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:39.085841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:39.085737 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:39.127843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:39.120334 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:39.127843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:39.120408 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:39.445486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:39.444535 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:48:39.445486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:39.444698 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:48:39.967137 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:39.965492 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:39.967137 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:39.965552 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:40.128337 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:40.128302 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:40.128650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:40.128604 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:40.154203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:40.151098 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:40.154203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:40.151149 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:40.915350 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:40.915317 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:48:40.915633 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:40.915604 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:48:41.003521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:40.995557 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:41.003521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:40.995636 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:41.141802 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:41.141763 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:41.142012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:41.141827 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:41.144096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:41.144068 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:41.144267 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:41.144243 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:41.489707 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:41.489668 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:48:41.508315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:41.508266 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:48:41.971484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:41.970592 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:41.971484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:41.970661 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:42.066484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:42.065849 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 05:48:42.066484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:42.065913 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 05:48:42.119485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:42.114815 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 05:48:42.119485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:42.114919 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 05:48:42.424423 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:42.424386 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:48:42.424756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:42.424727 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 05:48:43.004260 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:43.004223 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:43.004534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:43.004503 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 05:48:43.579089 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:43.579054 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:43.580663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:43.580617 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:43.582449 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:43.582424 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:48:43.601612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:43.598872 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:43.601612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:43.600138 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:43.603314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:43.603098 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:48:43.999614 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:43.999579 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:44.000229 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:44.000203 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:44.003670 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:44.003648 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:48:44.443899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:44.443870 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:44.444605 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:44.444360 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:44.447539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:44.447498 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:48:47.077606 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.077566 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6"] Oct 30 05:48:47.387543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.343891 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6"] Oct 30 05:48:47.387543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.344036 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.464484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.405343 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.589736 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.589645 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.599704 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.599660 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.604608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.604512 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.604818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.604801 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.658187 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.658097 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.658429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.658407 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.658645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.658625 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.658787 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.658769 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmfr\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-kube-api-access-8lmfr\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.687557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.687524 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8"] Oct 30 05:48:47.739824 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.739792 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.767539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.763354 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.767539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.763411 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.768671 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.768778 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.768828 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmfr\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-kube-api-access-8lmfr\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.768911 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.769012 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.769125 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.769157 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.769183 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.769627 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.769863 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.774509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.770105 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.801429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.800507 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.866704 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.866638 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmfr\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-kube-api-access-8lmfr\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.888483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.871970 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe543f-fbfb-4950-a940-8b25266ea5be-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.888483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.872042 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.888483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.872089 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6chn\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-kube-api-access-s6chn\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.888483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.872136 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.888483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.872169 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.888483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.872275 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.888483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.872364 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.888483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.872436 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.888483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.882767 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:47.945107 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.944996 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.946152 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.946097 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:47.954034 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.953977 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.985287 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.985373 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.985477 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.985538 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe543f-fbfb-4950-a940-8b25266ea5be-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.985577 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.985616 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6chn\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-kube-api-access-s6chn\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.985656 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.985688 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.985788 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.986275 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:47.986517 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.003010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.002962 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.016509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.010135 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8"] Oct 30 05:48:48.017347 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.017305 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe543f-fbfb-4950-a940-8b25266ea5be-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.035518 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.027626 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.058599 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.050478 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.061406 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.061338 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.075516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.067996 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.075516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.071093 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6chn\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-kube-api-access-s6chn\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.203539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.203194 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:48.217413 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:48.215723 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:49.232558 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:49.232506 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:49.639528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:49.625319 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:49.688613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:49.688563 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:48:49.734010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:49.733960 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:48:50.220494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:50.218009 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:50.856482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:50.839262 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:50.885013 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:50.878700 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:48:51.323852 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:51.323814 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7r6n" podUID="36b1cfca-d939-4a07-92cc-b5bc2520337b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 05:48:51.501501 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:48:51.471917 5072 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.96s" Oct 30 05:48:51.621087 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:51.621031 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:51.688835 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:51.688788 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:51.689135 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:51.689106 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:51.844485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:51.819429 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:48:52.696867 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:51.878203 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:52.698225 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:48:51.930741 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbe543f_fbfb_4950_a940_8b25266ea5be.slice/crio-bf7089b9afc868666baa602f7c1f1f4bc5f4b5c33489cc59003007fa4149ff78 WatchSource:0}: Error finding container bf7089b9afc868666baa602f7c1f1f4bc5f4b5c33489cc59003007fa4149ff78: Status 404 returned error can't find the container with id bf7089b9afc868666baa602f7c1f1f4bc5f4b5c33489cc59003007fa4149ff78 Oct 30 05:48:52.698383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:52.172822 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:52.698681 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:52.328529 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:52.698908 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:52.698877 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:52.717638 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:52.717589 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:48:53.015735 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.011150 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" event={"ID":"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a","Type":"ContainerStarted","Data":"fddddcec816c4665885afb56dc78c14335d659f17d4130bf275991e5e3734e28"} Oct 30 05:48:53.015735 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.011398 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6"] Oct 30 05:48:53.044404 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.011449 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8"] Oct 30 05:48:53.044404 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.028771 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:48:53.632517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.628137 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:48:53.666239 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.666202 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-q9b69-px5jl"] Oct 30 05:48:53.731705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.731673 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:53.744399 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.744372 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q9b69-px5jl"] Oct 30 05:48:53.779133 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.779104 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:48:53.813361 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.813332 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" event={"ID":"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a","Type":"ContainerStarted","Data":"83c6a223383342c8172709a641a524087332d6c65a97330203d813281fe10014"} Oct 30 05:48:53.867779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.867741 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" event={"ID":"3cbe543f-fbfb-4950-a940-8b25266ea5be","Type":"ContainerStarted","Data":"bf7089b9afc868666baa602f7c1f1f4bc5f4b5c33489cc59003007fa4149ff78"} Oct 30 05:48:53.965481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.961717 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-trusted-ca-bundle\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:53.965481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.961765 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpslj\" (UniqueName: \"kubernetes.io/projected/c892076a-fd0f-4633-8d4d-10f996f774ca-kube-api-access-xpslj\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:53.965481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.961839 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-service-ca-bundle\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:53.965481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:53.961961 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c892076a-fd0f-4633-8d4d-10f996f774ca-archives-path\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.072487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.067391 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-trusted-ca-bundle\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.072487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.067430 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpslj\" (UniqueName: \"kubernetes.io/projected/c892076a-fd0f-4633-8d4d-10f996f774ca-kube-api-access-xpslj\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.072487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.067501 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-service-ca-bundle\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.072487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.067570 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c892076a-fd0f-4633-8d4d-10f996f774ca-archives-path\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.072487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.068030 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c892076a-fd0f-4633-8d4d-10f996f774ca-archives-path\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.072487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.068916 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-service-ca-bundle\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.072487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.068916 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-trusted-ca-bundle\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.100742 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.100711 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpslj\" (UniqueName: \"kubernetes.io/projected/c892076a-fd0f-4633-8d4d-10f996f774ca-kube-api-access-xpslj\") pod \"periodic-gathering-q9b69-px5jl\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.303004 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.302926 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:48:54.630303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.630260 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q9b69-px5jl"] Oct 30 05:48:54.885534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.885425 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" event={"ID":"3cbe543f-fbfb-4950-a940-8b25266ea5be","Type":"ContainerStarted","Data":"1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55"} Oct 30 05:48:54.887832 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.887803 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q9b69-px5jl" event={"ID":"c892076a-fd0f-4633-8d4d-10f996f774ca","Type":"ContainerStarted","Data":"7572d28179bb75363f11130aad8e4baac6c7f6cef48010be27bd89b34970a30c"} Oct 30 05:48:54.887943 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.887843 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q9b69-px5jl" event={"ID":"c892076a-fd0f-4633-8d4d-10f996f774ca","Type":"ContainerStarted","Data":"165716c0dbb17fda68b844435394e531f0b6b1d49085c3d525aa91d99f7632b0"} Oct 30 05:48:54.994248 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.994192 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" podStartSLOduration=7.994171929 podStartE2EDuration="7.994171929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:54.934903058 +0000 UTC m=+1094.538592325" watchObservedRunningTime="2026-10-30 05:48:54.994171929 +0000 UTC m=+1094.597861189" Oct 30 05:48:54.996776 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:54.996733 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-q9b69-px5jl" podStartSLOduration=1.996718654 podStartE2EDuration="1.996718654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:54.991667205 +0000 UTC m=+1094.595356465" watchObservedRunningTime="2026-10-30 05:48:54.996718654 +0000 UTC m=+1094.600407916" Oct 30 05:48:55.222544 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:55.222009 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:48:55.233547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:55.233043 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:48:55.233547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:55.233218 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:48:55.233547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:55.233264 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:48:55.234173 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:55.234073 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:55.234173 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:55.234115 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 05:48:56.204987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:56.204949 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:56.205381 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:56.205012 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 05:48:56.222935 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:56.222895 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:48:56.223714 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:56.223679 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:48:57.205117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:57.205067 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:57.205665 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:57.205140 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 05:48:57.231485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:57.219849 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:48:57.231485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:57.219907 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:48:58.204879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:58.204848 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:58.205063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:58.204899 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 05:48:58.229485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:58.216815 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:48:58.229485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:58.216873 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:48:59.239485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:59.208786 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:59.242691 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:59.242653 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 05:48:59.267613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:59.232357 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:48:59.267889 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:48:59.267855 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:49:00.204936 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:00.204907 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:49:00.205172 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:00.205147 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 05:49:00.216847 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:00.216812 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:49:00.217105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:00.217077 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:49:00.840484 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:49:00.827014 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:49:01.205255 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:01.205213 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:49:01.205421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:01.205280 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 05:49:01.216677 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:01.216642 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 05:49:01.216846 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:01.216707 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 05:49:02.225482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:02.224974 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:49:02.225948 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:02.225937 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:49:02.230504 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:02.228056 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:49:02.284563 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:02.284510 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podStartSLOduration=15.284493611 podStartE2EDuration="15.284493611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:55.043630285 +0000 UTC m=+1094.647319590" watchObservedRunningTime="2026-10-30 05:49:02.284493611 +0000 UTC m=+1101.888182867" Oct 30 05:49:02.406334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:02.406300 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:49:02.406840 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:02.406821 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:49:02.408753 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:02.408729 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:49:16.664518 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:16.651537 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:17.723986 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:17.716184 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:17.725512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:17.725483 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:49:18.434105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:18.434074 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:49:18.654484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:18.623556 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:49:18.654484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:18.623618 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:18.871619 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:18.806542 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:49:18.871619 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:18.806611 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:18.872877 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:18.872377 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:19.185822 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:19.185696 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:19.492695 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:19.492594 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:20.505817 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:20.505587 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:20.515492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:20.507197 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:49:21.060548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:21.042901 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:21.318538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:21.303226 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:49:30.722241 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:49:30.722172 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:49:32.623877 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.623841 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz"] Oct 30 05:49:32.629918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.629896 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.659428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.659391 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz"] Oct 30 05:49:32.745105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.745062 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb610158-0784-4321-abee-9f48c8ba2bba-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.745105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.745105 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.745320 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.745289 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb610158-0784-4321-abee-9f48c8ba2bba-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.745438 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.745415 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.745514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.745493 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.745640 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.745621 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.745708 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.745684 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.745803 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.745784 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrntx\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-kube-api-access-lrntx\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.745853 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.745838 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.847683 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.847649 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrntx\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-kube-api-access-lrntx\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.847903 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.847706 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.847903 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.847762 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb610158-0784-4321-abee-9f48c8ba2bba-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.847903 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.847779 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.847903 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.847849 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb610158-0784-4321-abee-9f48c8ba2bba-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.848114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.847901 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.848114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.847958 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.848114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.848032 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.848114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.848085 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.848395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.848361 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.848641 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.848610 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb610158-0784-4321-abee-9f48c8ba2bba-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.848739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.848644 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.848739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.848663 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.848739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.848687 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.850745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.850718 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.850829 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.850744 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb610158-0784-4321-abee-9f48c8ba2bba-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.870429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.870395 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.870768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.870743 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrntx\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-kube-api-access-lrntx\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:32.948475 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:32.948382 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:33.279091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:33.279059 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz"] Oct 30 05:49:33.279613 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:49:33.279590 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb610158_0784_4321_abee_9f48c8ba2bba.slice/crio-922329c87737ed5d2970804b22819da3d65ddc6e34a04775158786e0d79e0982 WatchSource:0}: Error finding container 922329c87737ed5d2970804b22819da3d65ddc6e34a04775158786e0d79e0982: Status 404 returned error can't find the container with id 922329c87737ed5d2970804b22819da3d65ddc6e34a04775158786e0d79e0982 Oct 30 05:49:33.282689 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:33.282659 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:49:33.282795 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:33.282721 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:49:33.282795 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:33.282750 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:49:33.838991 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:33.838957 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" event={"ID":"fb610158-0784-4321-abee-9f48c8ba2bba","Type":"ContainerStarted","Data":"631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651"} Oct 30 05:49:33.839511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:33.839009 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" event={"ID":"fb610158-0784-4321-abee-9f48c8ba2bba","Type":"ContainerStarted","Data":"922329c87737ed5d2970804b22819da3d65ddc6e34a04775158786e0d79e0982"} Oct 30 05:49:33.948784 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:33.948690 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:34.951525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:34.951496 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:49:34.951944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:34.951544 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:49:35.949604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:35.949562 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:49:35.949818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:35.949651 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:49:36.949061 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:36.949022 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:49:36.949653 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:36.949083 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:49:37.952251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:37.951806 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:49:37.952251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:37.951867 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:49:38.951072 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:38.951031 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:49:38.951355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:38.951321 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:49:39.949326 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:39.949283 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:49:39.949850 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:39.949354 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 05:49:40.979625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:40.978407 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:40.979625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:40.979421 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:40.981032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:40.981006 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 05:49:41.042782 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:41.042729 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" podStartSLOduration=9.042716343 podStartE2EDuration="9.042716343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:33.944195682 +0000 UTC m=+1133.547884939" watchObservedRunningTime="2026-10-30 05:49:41.042716343 +0000 UTC m=+1140.646405599" Oct 30 05:49:44.714604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:44.714539 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:44.809488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:44.806571 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:45.587485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.586938 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c"] Oct 30 05:49:45.648483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.613740 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:45.742485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.646476 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c"] Oct 30 05:49:45.743766 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.743735 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"openshift-service-ca.crt\"" Oct 30 05:49:45.788557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.743714 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"kube-root-ca.crt\"" Oct 30 05:49:45.889122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.744078 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"distinct-2-w5-be78f2-kserve-self-signed-certs\"" Oct 30 05:49:45.889122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.744818 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"default-dockercfg-nchgr\"" Oct 30 05:49:45.889122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.755907 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tmp-dir\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:45.889122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.816094 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tls-certs\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:45.889122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.816171 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvf48\" (UniqueName: \"kubernetes.io/projected/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-kube-api-access-xvf48\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:45.889122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.816210 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-home\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:45.889122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.816357 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-dshm\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:45.889122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.816404 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-model-cache\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.237682 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.921431 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tmp-dir\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.252627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:45.931715 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tmp-dir\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.289487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.286505 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tls-certs\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.299435 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.299402 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvf48\" (UniqueName: \"kubernetes.io/projected/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-kube-api-access-xvf48\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.299722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.299700 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-home\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.350775 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.350740 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-dshm\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.377130 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.377096 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-model-cache\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.428585 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.303819 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-home\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.428585 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.396374 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-model-cache\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.680498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.635774 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-dshm\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.692492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.681484 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvf48\" (UniqueName: \"kubernetes.io/projected/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-kube-api-access-xvf48\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.740500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.682050 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tls-certs\") pod \"distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:46.840480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:46.838047 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:47.860280 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:49:47.860216 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51afb6ee_e6d5_4f7f_a6c8_0b0c8bd9fac9.slice/crio-dc9b8e2904528737211922d1dcc6083148da06e5c775ba612e77ad919fd74771 WatchSource:0}: Error finding container dc9b8e2904528737211922d1dcc6083148da06e5c775ba612e77ad919fd74771: Status 404 returned error can't find the container with id dc9b8e2904528737211922d1dcc6083148da06e5c775ba612e77ad919fd74771 Oct 30 05:49:47.997484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:47.980063 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c"] Oct 30 05:49:48.465501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:48.465040 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" event={"ID":"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9","Type":"ContainerStarted","Data":"ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf"} Oct 30 05:49:48.465501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:48.465099 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" event={"ID":"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9","Type":"ContainerStarted","Data":"dc9b8e2904528737211922d1dcc6083148da06e5c775ba612e77ad919fd74771"} Oct 30 05:49:48.517856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:48.517792 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podStartSLOduration=3.517769815 podStartE2EDuration="3.517769815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:48.506429396 +0000 UTC m=+1148.110118658" watchObservedRunningTime="2026-10-30 05:49:48.517769815 +0000 UTC m=+1148.121459073" Oct 30 05:49:48.868700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:48.868666 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:56.869233 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:56.869198 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:56.882161 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:56.882136 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:57.598104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:57.598074 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:49:58.048393 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.048359 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8871-95a03f"] Oct 30 05:49:58.052420 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.052400 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8871-95a03f" Oct 30 05:49:58.067045 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.067020 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bnq\"" Oct 30 05:49:58.069649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.068878 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8871-95a03f"] Oct 30 05:49:58.160081 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.160044 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8gh7\" (UniqueName: \"kubernetes.io/projected/f14fb99e-eecb-4f0e-9557-e2ad4758ab0a-kube-api-access-x8gh7\") pod \"test-curl-8871-95a03f\" (UID: \"f14fb99e-eecb-4f0e-9557-e2ad4758ab0a\") " pod="opendatahub/test-curl-8871-95a03f" Oct 30 05:49:58.261348 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.261314 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8gh7\" (UniqueName: \"kubernetes.io/projected/f14fb99e-eecb-4f0e-9557-e2ad4758ab0a-kube-api-access-x8gh7\") pod \"test-curl-8871-95a03f\" (UID: \"f14fb99e-eecb-4f0e-9557-e2ad4758ab0a\") " pod="opendatahub/test-curl-8871-95a03f" Oct 30 05:49:58.284916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.284884 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8gh7\" (UniqueName: \"kubernetes.io/projected/f14fb99e-eecb-4f0e-9557-e2ad4758ab0a-kube-api-access-x8gh7\") pod \"test-curl-8871-95a03f\" (UID: \"f14fb99e-eecb-4f0e-9557-e2ad4758ab0a\") " pod="opendatahub/test-curl-8871-95a03f" Oct 30 05:49:58.365326 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.365296 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8871-95a03f" Oct 30 05:49:58.727966 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.727915 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8871-95a03f"] Oct 30 05:49:58.729131 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:49:58.729103 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf14fb99e_eecb_4f0e_9557_e2ad4758ab0a.slice/crio-a3bc974951cec9868b3045132b2baf3c2dffece8a234eee0c7ed4d84e731295f WatchSource:0}: Error finding container a3bc974951cec9868b3045132b2baf3c2dffece8a234eee0c7ed4d84e731295f: Status 404 returned error can't find the container with id a3bc974951cec9868b3045132b2baf3c2dffece8a234eee0c7ed4d84e731295f Oct 30 05:49:58.731312 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:58.731293 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:49:59.639556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:49:59.639512 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8871-95a03f" event={"ID":"f14fb99e-eecb-4f0e-9557-e2ad4758ab0a","Type":"ContainerStarted","Data":"a3bc974951cec9868b3045132b2baf3c2dffece8a234eee0c7ed4d84e731295f"} Oct 30 05:50:00.726529 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:50:00.720321 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:50:01.660111 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:01.660071 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8871-95a03f" event={"ID":"f14fb99e-eecb-4f0e-9557-e2ad4758ab0a","Type":"ContainerStarted","Data":"d99a09a57afe547539e9ac2058829f60af0637c4d85e81a9e910337e357f106d"} Oct 30 05:50:01.698285 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:01.698226 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8871-95a03f" podStartSLOduration=1.365791672 podStartE2EDuration="3.698210924s" totalImagesPullingTime="2.332419252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:01.69697054 +0000 UTC m=+1161.300659803" watchObservedRunningTime="2026-10-30 05:50:01.698210924 +0000 UTC m=+1161.301900198" Oct 30 05:50:02.636698 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:02.636665 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8871-95a03f"] Oct 30 05:50:02.642732 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:02.642703 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8871-95a03f"] Oct 30 05:50:03.031789 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.031703 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2"] Oct 30 05:50:03.033148 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.033118 5072 cpu_manager.go:414] "containerMap: removing container" podUID="f14fb99e-eecb-4f0e-9557-e2ad4758ab0a" containerName="test-curl-8871-95a03f" Oct 30 05:50:03.033148 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.033147 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14fb99e-eecb-4f0e-9557-e2ad4758ab0a" containerName="test-curl-8871-95a03f" Oct 30 05:50:03.033347 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.033322 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14fb99e-eecb-4f0e-9557-e2ad4758ab0a" containerName="test-curl-8871-95a03f" Oct 30 05:50:03.037862 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.037839 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.071745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.071712 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2"] Oct 30 05:50:03.210615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.210563 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae92881-1919-4d77-813e-d25ccd1b31c1-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.210841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.210628 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.210841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.210698 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.210841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.210747 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.211027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.210868 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.211027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.210907 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqkj\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-kube-api-access-ktqkj\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.211027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.210936 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.211182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.211060 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.211182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.211122 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.314433 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.314330 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqkj\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-kube-api-access-ktqkj\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.314433 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.314387 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.314700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.314514 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.314700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.314570 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.314700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.314651 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae92881-1919-4d77-813e-d25ccd1b31c1-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.314700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.314697 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.314935 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.314758 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.314935 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.314806 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.314935 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.314925 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.315416 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.315369 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.315816 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.315792 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.316168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.316145 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.316582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.316554 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae92881-1919-4d77-813e-d25ccd1b31c1-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.316776 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.316754 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.319978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.319954 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.320226 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.320201 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.360681 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.360647 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqkj\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-kube-api-access-ktqkj\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.362471 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.362389 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.654243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.654207 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:03.681889 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:03.681854 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8871-95a03f" podUID="f14fb99e-eecb-4f0e-9557-e2ad4758ab0a" containerName="test-curl-8871-95a03f" containerID="cri-o://d99a09a57afe547539e9ac2058829f60af0637c4d85e81a9e910337e357f106d" gracePeriod=2 Oct 30 05:50:03.699075 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:50:03.699017 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8871-95a03f\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="f14fb99e-eecb-4f0e-9557-e2ad4758ab0a" pod="opendatahub/test-curl-8871-95a03f" Oct 30 05:50:04.009622 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:04.009590 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2"] Oct 30 05:50:04.010534 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:50:04.010493 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae92881_1919_4d77_813e_d25ccd1b31c1.slice/crio-a4a822556d4a998f5d0cc9f563361f19ffdc3f8b5e87d4444c3d7352065625af WatchSource:0}: Error finding container a4a822556d4a998f5d0cc9f563361f19ffdc3f8b5e87d4444c3d7352065625af: Status 404 returned error can't find the container with id a4a822556d4a998f5d0cc9f563361f19ffdc3f8b5e87d4444c3d7352065625af Oct 30 05:50:04.013652 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:04.013605 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:04.013730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:04.013719 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:04.013777 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:04.013762 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:04.760356 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:04.760317 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" event={"ID":"bae92881-1919-4d77-813e-d25ccd1b31c1","Type":"ContainerStarted","Data":"524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b"} Oct 30 05:50:04.760881 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:04.760398 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" event={"ID":"bae92881-1919-4d77-813e-d25ccd1b31c1","Type":"ContainerStarted","Data":"a4a822556d4a998f5d0cc9f563361f19ffdc3f8b5e87d4444c3d7352065625af"} Oct 30 05:50:04.834028 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:04.833565 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podStartSLOduration=2.833543578 podStartE2EDuration="2.833543578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:04.81701129 +0000 UTC m=+1164.420700559" watchObservedRunningTime="2026-10-30 05:50:04.833543578 +0000 UTC m=+1164.437232838" Oct 30 05:50:05.655158 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:05.655127 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:05.691534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:05.690769 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:50:05.691534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:05.690847 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:50:05.773705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:05.773678 5072 generic.go:356] "Generic (PLEG): container finished" podID="f14fb99e-eecb-4f0e-9557-e2ad4758ab0a" containerID="d99a09a57afe547539e9ac2058829f60af0637c4d85e81a9e910337e357f106d" exitCode=137 Oct 30 05:50:05.991065 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:05.991040 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8871-95a03f" Oct 30 05:50:06.042031 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:06.041992 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14fb99e-eecb-4f0e-9557-e2ad4758ab0a-kube-api-access-x8gh7\" (UniqueName: \"kubernetes.io/projected/f14fb99e-eecb-4f0e-9557-e2ad4758ab0a-kube-api-access-x8gh7\") pod \"f14fb99e-eecb-4f0e-9557-e2ad4758ab0a\" (UID: \"f14fb99e-eecb-4f0e-9557-e2ad4758ab0a\") " Oct 30 05:50:06.044956 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:06.044925 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14fb99e-eecb-4f0e-9557-e2ad4758ab0a-kube-api-access-x8gh7" pod "f14fb99e-eecb-4f0e-9557-e2ad4758ab0a" (UID: "f14fb99e-eecb-4f0e-9557-e2ad4758ab0a"). InnerVolumeSpecName "kube-api-access-x8gh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:06.143911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:06.143876 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8gh7\" (UniqueName: \"kubernetes.io/projected/f14fb99e-eecb-4f0e-9557-e2ad4758ab0a-kube-api-access-x8gh7\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:50:06.546479 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:06.545772 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14fb99e-eecb-4f0e-9557-e2ad4758ab0a" path="/var/lib/kubelet/pods/f14fb99e-eecb-4f0e-9557-e2ad4758ab0a/volumes" Oct 30 05:50:06.655761 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:06.655720 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:50:06.655959 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:06.655792 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:50:06.782438 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:06.782409 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8871-95a03f" Oct 30 05:50:06.782936 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:06.782448 5072 scope.go:122] "RemoveContainer" containerID="d99a09a57afe547539e9ac2058829f60af0637c4d85e81a9e910337e357f106d" Oct 30 05:50:07.655590 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:07.654665 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:50:07.655590 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:07.654729 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:50:08.655243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:08.655205 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:50:08.655704 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:08.655274 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:50:09.655411 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:09.655364 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:50:09.655884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:09.655417 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:50:10.657841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:10.657793 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:50:10.658625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:10.658574 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:50:11.704291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:11.704244 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:50:11.704908 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:11.704874 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:50:12.691486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:12.687834 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:50:12.691486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:12.687907 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 05:50:12.853680 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:12.852905 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:50:13.313731 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:13.313683 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.598658 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:13.598551 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:50:13.951309 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:13.951260 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.967402 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:13.967370 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:50:14.349541 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:14.324007 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:50:14.349541 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:14.324936 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:50:14.692280 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:14.608607 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.720494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:14.720203 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.887910 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:14.887861 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:14.913549 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:14.888166 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.053861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:15.053749 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:50:15.138486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:15.130979 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:50:15.595700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:15.595655 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:15.596514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:15.596450 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.622516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:15.621958 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:50:15.706487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:15.670279 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:50:16.207486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.115962 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.209617 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.162515 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:16.209928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.209893 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.210623 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.162580 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" podUID="df4f6519-1409-4d13-b154-a1313e04c7fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.332615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.332576 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb"] Oct 30 05:50:16.358492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.358416 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.453920 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.453037 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6a881dd9-openshift-default-dockercfg-98dh2\"" Oct 30 05:50:16.500642 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.500561 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb"] Oct 30 05:50:16.546487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.539745 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-socket\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.546487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.539852 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-token\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.546487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.539891 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-podinfo\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.546487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.539953 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-data\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.546487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.540087 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbeb647-4db1-47e0-8abb-88619bb76907-istiod-ca-cert\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.546487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.540118 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-credential-socket\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.546487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.540188 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-certs\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.546487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.540215 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-envoy\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.546487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.540244 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vk6\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-kube-api-access-f9vk6\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.625673 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.625625 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.699515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.653695 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-podinfo\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.699515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.653796 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-data\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.699515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.653957 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbeb647-4db1-47e0-8abb-88619bb76907-istiod-ca-cert\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.699515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.653988 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-credential-socket\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.699515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.654065 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-certs\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.699515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.654088 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-envoy\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.699515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.654118 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vk6\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-kube-api-access-f9vk6\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.699515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.654182 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-socket\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.699515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.654286 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-token\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.748486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.731504 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-certs\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.795571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.761208 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:50:16.888482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.764777 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-data\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.888482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.769084 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbeb647-4db1-47e0-8abb-88619bb76907-istiod-ca-cert\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.888482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.783774 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-credential-socket\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.888482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.786674 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vk6\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-kube-api-access-f9vk6\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.888482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.811901 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-token\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.888482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.822838 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-socket\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.954488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.950484 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-podinfo\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:16.962926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:16.962853 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-envoy\") pod \"e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:17.222884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:17.191532 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:17.236759 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:17.226601 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:17.262615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:17.262575 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:17.276493 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:17.262861 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:17.629484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:17.627538 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:50:18.059484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.041772 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:18.075485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.067551 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:18.186484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.067734 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:18.186484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.096287 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:18.186484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.175296 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 05:50:18.186484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.175347 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:50:18.312107 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.296263 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:50:18.603388 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.526331 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:18.603388 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.526410 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:18.695503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.674887 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:18.756187 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.756147 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:18.756485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:18.756434 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:19.192598 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:19.175025 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:19.540538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:19.502851 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:50:19.681480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:19.680568 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:50:19.712608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:19.712564 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:19.712886 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:19.712863 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:19.713064 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:19.713036 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:19.833554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:19.802833 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb"] Oct 30 05:50:19.859485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:19.838242 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" event={"ID":"cbbeb647-4db1-47e0-8abb-88619bb76907","Type":"ContainerStarted","Data":"380b08ba7eb7e4be583061d11627476ebf6c8e43228c4cbed225cd4a1763dbb1"} Oct 30 05:50:20.062657 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:20.062616 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:50:20.527482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:20.527418 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:50:20.683674 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:20.683616 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:20.703974 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:20.703938 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:50:20.769984 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:20.769956 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:50:20.850956 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:20.850872 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" event={"ID":"cbbeb647-4db1-47e0-8abb-88619bb76907","Type":"ContainerStarted","Data":"ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211"} Oct 30 05:50:20.937806 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:20.937733 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" podStartSLOduration=4.937714466 podStartE2EDuration="4.937714466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:20.937339534 +0000 UTC m=+1180.541028810" watchObservedRunningTime="2026-10-30 05:50:20.937714466 +0000 UTC m=+1180.541403724" Oct 30 05:50:21.199512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.198564 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:21.207492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.205568 5072 patch_prober.go:29] interesting pod/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 05:50:21.207492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.205629 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" podUID="cbbeb647-4db1-47e0-8abb-88619bb76907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 05:50:21.574918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.574832 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5"] Oct 30 05:50:21.583822 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.583760 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.602085 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.602056 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3cc280cf-gw-openshift-default-dockercfg-bbqpp\"" Oct 30 05:50:21.629884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.629845 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae030c-e547-4310-95b5-5db3086fed24-istiod-ca-cert\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.630068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.629937 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-istio-token\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.630068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.629976 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae030c-e547-4310-95b5-5db3086fed24-istio-podinfo\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.630068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.630049 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-credential-socket\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.630233 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.630207 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-envoy\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.630360 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.630295 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-certs\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.630360 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.630323 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-data\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.630360 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.630346 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfcz\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-kube-api-access-2qfcz\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.630561 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.630398 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-socket\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.630561 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.630532 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5"] Oct 30 05:50:21.731279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.731245 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-envoy\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.731791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.731338 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-certs\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.731791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.731364 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-data\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.731791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.731392 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfcz\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-kube-api-access-2qfcz\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.731791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.731448 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-socket\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.731791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.731527 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae030c-e547-4310-95b5-5db3086fed24-istiod-ca-cert\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.731791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.731598 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-istio-token\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.731791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.731634 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae030c-e547-4310-95b5-5db3086fed24-istio-podinfo\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.731791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.731684 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-credential-socket\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.732401 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.732347 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae030c-e547-4310-95b5-5db3086fed24-istiod-ca-cert\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.732576 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.732497 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-data\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.732649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.732601 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-credential-socket\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.732868 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.732846 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-socket\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.733034 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.733015 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-certs\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.734633 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.734609 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-envoy\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.734730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.734705 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae030c-e547-4310-95b5-5db3086fed24-istio-podinfo\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.792263 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.791546 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-istio-token\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.808483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.795724 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfcz\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-kube-api-access-2qfcz\") pod \"e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:21.914994 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:21.914962 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:22.198542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.197720 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:22.489387 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.489356 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5"] Oct 30 05:50:22.491002 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:50:22.490974 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceae030c_e547_4310_95b5_5db3086fed24.slice/crio-b771b135eb4722fda0d43b2625355c853d302dca7718a47a32793524866989a1 WatchSource:0}: Error finding container b771b135eb4722fda0d43b2625355c853d302dca7718a47a32793524866989a1: Status 404 returned error can't find the container with id b771b135eb4722fda0d43b2625355c853d302dca7718a47a32793524866989a1 Oct 30 05:50:22.493676 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.493643 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:22.493757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.493712 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:22.493757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.493742 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:22.923785 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.923748 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" event={"ID":"ceae030c-e547-4310-95b5-5db3086fed24","Type":"ContainerStarted","Data":"f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e"} Oct 30 05:50:22.924288 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.923810 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" event={"ID":"ceae030c-e547-4310-95b5-5db3086fed24","Type":"ContainerStarted","Data":"b771b135eb4722fda0d43b2625355c853d302dca7718a47a32793524866989a1"} Oct 30 05:50:22.924288 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.923999 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:22.925123 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.925101 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:50:22.985788 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:22.985741 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" podStartSLOduration=1.985728717 podStartE2EDuration="1.985728717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:22.981166859 +0000 UTC m=+1182.584856116" watchObservedRunningTime="2026-10-30 05:50:22.985728717 +0000 UTC m=+1182.589417974" Oct 30 05:50:23.918891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:23.918851 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:23.924741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:23.924708 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:23.933383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:23.933357 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:23.934052 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:23.934028 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:50:30.713222 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:50:30.713186 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:50:31.077551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:31.077443 5072 generic.go:356] "Generic (PLEG): container finished" podID="c892076a-fd0f-4633-8d4d-10f996f774ca" containerID="7572d28179bb75363f11130aad8e4baac6c7f6cef48010be27bd89b34970a30c" exitCode=0 Oct 30 05:50:31.077551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:31.077516 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q9b69-px5jl" event={"ID":"c892076a-fd0f-4633-8d4d-10f996f774ca","Type":"ContainerDied","Data":"7572d28179bb75363f11130aad8e4baac6c7f6cef48010be27bd89b34970a30c"} Oct 30 05:50:32.275911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.275885 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:50:32.410979 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.410944 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c892076a-fd0f-4633-8d4d-10f996f774ca-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c892076a-fd0f-4633-8d4d-10f996f774ca-archives-path\") pod \"c892076a-fd0f-4633-8d4d-10f996f774ca\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " Oct 30 05:50:32.411164 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.411011 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c892076a-fd0f-4633-8d4d-10f996f774ca-kube-api-access-xpslj\" (UniqueName: \"kubernetes.io/projected/c892076a-fd0f-4633-8d4d-10f996f774ca-kube-api-access-xpslj\") pod \"c892076a-fd0f-4633-8d4d-10f996f774ca\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " Oct 30 05:50:32.411252 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.411235 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-trusted-ca-bundle\") pod \"c892076a-fd0f-4633-8d4d-10f996f774ca\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " Oct 30 05:50:32.411304 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.411288 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-service-ca-bundle\") pod \"c892076a-fd0f-4633-8d4d-10f996f774ca\" (UID: \"c892076a-fd0f-4633-8d4d-10f996f774ca\") " Oct 30 05:50:32.411355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.411315 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c892076a-fd0f-4633-8d4d-10f996f774ca-archives-path" pod "c892076a-fd0f-4633-8d4d-10f996f774ca" (UID: "c892076a-fd0f-4633-8d4d-10f996f774ca"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:50:32.411689 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.411665 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-trusted-ca-bundle" pod "c892076a-fd0f-4633-8d4d-10f996f774ca" (UID: "c892076a-fd0f-4633-8d4d-10f996f774ca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:32.411851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.411772 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-service-ca-bundle" pod "c892076a-fd0f-4633-8d4d-10f996f774ca" (UID: "c892076a-fd0f-4633-8d4d-10f996f774ca"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:32.412256 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.412236 5072 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-trusted-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:50:32.412315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.412263 5072 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c892076a-fd0f-4633-8d4d-10f996f774ca-service-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:50:32.412315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.412280 5072 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c892076a-fd0f-4633-8d4d-10f996f774ca-archives-path\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:50:32.413572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.413549 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c892076a-fd0f-4633-8d4d-10f996f774ca-kube-api-access-xpslj" pod "c892076a-fd0f-4633-8d4d-10f996f774ca" (UID: "c892076a-fd0f-4633-8d4d-10f996f774ca"). InnerVolumeSpecName "kube-api-access-xpslj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:32.522487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.517667 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpslj\" (UniqueName: \"kubernetes.io/projected/c892076a-fd0f-4633-8d4d-10f996f774ca-kube-api-access-xpslj\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:50:32.895077 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.895046 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn"] Oct 30 05:50:32.897035 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.895802 5072 cpu_manager.go:414] "containerMap: removing container" podUID="c892076a-fd0f-4633-8d4d-10f996f774ca" containerName="insights-gathering" Oct 30 05:50:32.897035 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.895825 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c892076a-fd0f-4633-8d4d-10f996f774ca" containerName="insights-gathering" Oct 30 05:50:32.897035 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.896086 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c892076a-fd0f-4633-8d4d-10f996f774ca" containerName="insights-gathering" Oct 30 05:50:32.900280 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.900262 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:32.938023 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.937953 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn"] Oct 30 05:50:32.945734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.945605 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc"] Oct 30 05:50:32.952956 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.952906 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:32.970576 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:32.970532 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc"] Oct 30 05:50:33.026916 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.026879 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.027096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.026943 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.027096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027050 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.027295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027171 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.027295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027203 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.027295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027238 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117dc795-66d4-478c-9ff7-b23b056a4d48-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.027513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027299 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.027513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027336 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.027513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027364 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.027513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027420 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9wcw\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-kube-api-access-r9wcw\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.027748 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027541 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.027748 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027691 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.027748 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027717 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnvnc\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-kube-api-access-pnvnc\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.027748 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027741 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.027967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027851 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.027967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027895 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.027967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027927 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.027967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.027955 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.092526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.092496 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q9b69-px5jl" Oct 30 05:50:33.092526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.092511 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q9b69-px5jl" event={"ID":"c892076a-fd0f-4633-8d4d-10f996f774ca","Type":"ContainerDied","Data":"165716c0dbb17fda68b844435394e531f0b6b1d49085c3d525aa91d99f7632b0"} Oct 30 05:50:33.092718 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.092567 5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="165716c0dbb17fda68b844435394e531f0b6b1d49085c3d525aa91d99f7632b0" Oct 30 05:50:33.129187 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129159 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117dc795-66d4-478c-9ff7-b23b056a4d48-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.129391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129214 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.129391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129250 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.129391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129275 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.129391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129332 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wcw\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-kube-api-access-r9wcw\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.129721 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129446 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.129721 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129509 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.129721 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129532 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnvnc\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-kube-api-access-pnvnc\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.129721 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129562 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.129721 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129693 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.129721 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129700 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.129983 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129742 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.130025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129991 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117dc795-66d4-478c-9ff7-b23b056a4d48-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.130025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.129994 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.130025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130020 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.130156 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130051 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.130156 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130058 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.130249 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130150 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.130249 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130202 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.130249 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130229 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.130730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130307 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.130730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130313 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.130730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130365 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.130730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130598 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.130990 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130866 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.130990 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.130925 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.131188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.131168 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.131441 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.131421 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.133351 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.133326 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.133507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.133349 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.133579 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.133507 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.134001 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.133984 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.154577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.154515 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.155327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.155298 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.155442 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.155382 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wcw\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-kube-api-access-r9wcw\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.155442 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.155415 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnvnc\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-kube-api-access-pnvnc\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.214481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.214442 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:33.281088 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.280700 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:33.538512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.538478 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn"] Oct 30 05:50:33.539739 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:50:33.539711 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117dc795_66d4_478c_9ff7_b23b056a4d48.slice/crio-a4a4840eed66dd32d287e64761391aa167628920349b26b1dccef826db13f805 WatchSource:0}: Error finding container a4a4840eed66dd32d287e64761391aa167628920349b26b1dccef826db13f805: Status 404 returned error can't find the container with id a4a4840eed66dd32d287e64761391aa167628920349b26b1dccef826db13f805 Oct 30 05:50:33.542904 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.542866 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:33.543001 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.542950 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:33.543001 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.542995 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:33.615487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.615445 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc"] Oct 30 05:50:33.616534 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:50:33.616513 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe8a4c8_f5f3_4dcb_a3a9_e9e8ef2723ad.slice/crio-b7cd074e64ed436f486f7cf57936146770a455889fe1afed97568503e9e695a3 WatchSource:0}: Error finding container b7cd074e64ed436f486f7cf57936146770a455889fe1afed97568503e9e695a3: Status 404 returned error can't find the container with id b7cd074e64ed436f486f7cf57936146770a455889fe1afed97568503e9e695a3 Oct 30 05:50:33.619751 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.619721 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:33.619856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.619796 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:33.620046 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:33.619857 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:50:34.155248 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.155205 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" event={"ID":"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad","Type":"ContainerStarted","Data":"4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a"} Oct 30 05:50:34.169543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.169504 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" event={"ID":"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad","Type":"ContainerStarted","Data":"b7cd074e64ed436f486f7cf57936146770a455889fe1afed97568503e9e695a3"} Oct 30 05:50:34.179668 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.179629 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" event={"ID":"117dc795-66d4-478c-9ff7-b23b056a4d48","Type":"ContainerStarted","Data":"71e21cfe4e48d19b84e0ad35614974e470ce0d978bf8894cb987bc5aa3801355"} Oct 30 05:50:34.179926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.179905 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" event={"ID":"117dc795-66d4-478c-9ff7-b23b056a4d48","Type":"ContainerStarted","Data":"a4a4840eed66dd32d287e64761391aa167628920349b26b1dccef826db13f805"} Oct 30 05:50:34.215432 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.215076 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:34.217659 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.216537 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:50:34.217659 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.216595 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:50:34.238420 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.238342 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podStartSLOduration=2.238321988 podStartE2EDuration="2.238321988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:34.230798049 +0000 UTC m=+1193.834487315" watchObservedRunningTime="2026-10-30 05:50:34.238321988 +0000 UTC m=+1193.842011247" Oct 30 05:50:34.288481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.284568 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:34.288481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.285507 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 05:50:34.288481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.285543 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 05:50:34.288481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:34.286251 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podStartSLOduration=2.286238062 podStartE2EDuration="2.286238062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:34.28032455 +0000 UTC m=+1193.884013820" watchObservedRunningTime="2026-10-30 05:50:34.286238062 +0000 UTC m=+1193.889927376" Oct 30 05:50:35.220924 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.220881 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:50:35.221313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.221281 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:50:35.282221 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.282190 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 05:50:35.282444 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.282423 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 05:50:35.705226 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.705196 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k"] Oct 30 05:50:35.716485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.714312 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:35.752075 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.752039 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6a881dd9-serving-cert\"" Oct 30 05:50:35.796991 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.795683 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k"] Oct 30 05:50:35.803119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.802312 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0956524a-69aa-4f07-bf32-6511f39615fd-openshift-service-ca\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:35.803119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.802426 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqjk\" (UniqueName: \"kubernetes.io/projected/0956524a-69aa-4f07-bf32-6511f39615fd-kube-api-access-gmqjk\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:35.803119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.802585 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-metrics-certs\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:35.803119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.802688 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:35.903878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.903846 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-metrics-certs\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:35.904197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.904176 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:35.904386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.904367 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0956524a-69aa-4f07-bf32-6511f39615fd-openshift-service-ca\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:35.904599 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.904581 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqjk\" (UniqueName: \"kubernetes.io/projected/0956524a-69aa-4f07-bf32-6511f39615fd-kube-api-access-gmqjk\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:35.904831 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:50:35.904781 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-serving-cert: secret "maas-api-e2e-ait-6a881dd9-serving-cert" not found Oct 30 05:50:35.904897 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:50:35.904858 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls podName:0956524a-69aa-4f07-bf32-6511f39615fd nodeName:}" failed. No retries permitted until 2026-10-30 05:50:36.404837363 +0000 UTC m=+1196.008526619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls") pod "maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" (UID: "0956524a-69aa-4f07-bf32-6511f39615fd") : secret "maas-api-e2e-ait-6a881dd9-serving-cert" not found Oct 30 05:50:35.906104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:35.906082 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0956524a-69aa-4f07-bf32-6511f39615fd-openshift-service-ca\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:36.054539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:36.035020 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-metrics-certs\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:36.094486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:36.091175 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqjk\" (UniqueName: \"kubernetes.io/projected/0956524a-69aa-4f07-bf32-6511f39615fd-kube-api-access-gmqjk\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:36.234612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:36.221893 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:50:36.234612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:36.221968 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:50:36.282496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:36.281950 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 05:50:36.282496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:36.282014 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 05:50:36.469497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:36.469027 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:36.479247 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:36.479218 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls\") pod \"maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:36.712483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:36.709827 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:37.280930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:37.256606 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:50:37.280930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:37.256678 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:50:37.285444 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:37.283400 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k"] Oct 30 05:50:37.285444 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:37.283684 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 05:50:37.285444 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:37.283753 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 05:50:38.229681 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:38.216107 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:50:38.229681 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:38.216198 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:50:38.288289 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:38.288143 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 05:50:38.288289 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:38.288199 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 05:50:38.655481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:38.651054 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" event={"ID":"0956524a-69aa-4f07-bf32-6511f39615fd","Type":"ContainerStarted","Data":"638738fb1f2288062b01941f673b4def890ed9865950a338d968b7ed0560242e"} Oct 30 05:50:39.216494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:39.215818 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:50:39.216494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:39.215895 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:50:39.281891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:39.281848 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 05:50:39.282387 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:39.281904 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 05:50:39.577631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:39.577496 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" event={"ID":"0956524a-69aa-4f07-bf32-6511f39615fd","Type":"ContainerStarted","Data":"a9f11733051eb47820cd1ebb062930bc8a76b8830d6359d91ae6c8c84e29fa8f"} Oct 30 05:50:39.578109 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:39.578088 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:40.235486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.231845 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:50:40.235486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.231906 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:50:40.282349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.281506 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 05:50:40.282349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.281582 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 05:50:40.436289 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.436130 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" podStartSLOduration=4.729743718 podStartE2EDuration="5.436107859s" totalImagesPullingTime="706.364141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:39.642556441 +0000 UTC m=+1199.246245727" watchObservedRunningTime="2026-10-30 05:50:40.436107859 +0000 UTC m=+1200.039797114" Oct 30 05:50:40.440408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.440377 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k"] Oct 30 05:50:40.445056 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.445036 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.461651 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.460706 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3cc280cf-serving-cert\"" Oct 30 05:50:40.469498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.469153 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k"] Oct 30 05:50:40.562533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.561740 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-metrics-certs\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.562533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.561891 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.562533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.562042 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wzgg\" (UniqueName: \"kubernetes.io/projected/1951d6e6-b2c1-426d-8b70-2128fb7407a8-kube-api-access-7wzgg\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.562533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.562108 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1951d6e6-b2c1-426d-8b70-2128fb7407a8-openshift-service-ca\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.674485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.673908 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1951d6e6-b2c1-426d-8b70-2128fb7407a8-openshift-service-ca\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.674485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.674137 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-metrics-certs\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.674485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.674358 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.674766 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.674592 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wzgg\" (UniqueName: \"kubernetes.io/projected/1951d6e6-b2c1-426d-8b70-2128fb7407a8-kube-api-access-7wzgg\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.693482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.675881 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1951d6e6-b2c1-426d-8b70-2128fb7407a8-openshift-service-ca\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.730583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.730079 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3cc280cf-serving-cert\"" Oct 30 05:50:40.733437 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:50:40.732371 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-serving-cert: secret "maas-api-e2e-migrate-3cc280cf-serving-cert" not found Oct 30 05:50:40.733437 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:50:40.732496 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls podName:1951d6e6-b2c1-426d-8b70-2128fb7407a8 nodeName:}" failed. No retries permitted until 2026-10-30 05:50:41.232470592 +0000 UTC m=+1200.836159841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls") pod "maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" (UID: "1951d6e6-b2c1-426d-8b70-2128fb7407a8") : secret "maas-api-e2e-migrate-3cc280cf-serving-cert" not found Oct 30 05:50:40.765498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.764125 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-metrics-certs\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.797894 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.790897 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wzgg\" (UniqueName: \"kubernetes.io/projected/1951d6e6-b2c1-426d-8b70-2128fb7407a8-kube-api-access-7wzgg\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:40.961192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.961158 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:50:40.962063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:40.962042 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 05:50:41.215035 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:41.214939 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:50:41.215211 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:41.215036 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 05:50:41.281472 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:41.281426 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 05:50:41.281637 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:41.281530 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 05:50:41.309945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:41.309904 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:41.314651 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:41.314620 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls\") pod \"maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:41.408668 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:41.408633 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:42.052484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.044693 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k"] Oct 30 05:50:42.639933 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.638687 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" event={"ID":"1951d6e6-b2c1-426d-8b70-2128fb7407a8","Type":"ContainerStarted","Data":"6320071def8eefa9a1071f5ee7f52812884205924552b8ffd8df0bf58a0bf47b"} Oct 30 05:50:42.639933 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.638751 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" event={"ID":"1951d6e6-b2c1-426d-8b70-2128fb7407a8","Type":"ContainerStarted","Data":"185f8912e20f322ee9cbe67a142a50905c65b1c2d2b6b9429d9c911b25d36af0"} Oct 30 05:50:42.639933 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.639754 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:50:42.667726 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.667695 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:42.668087 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.668068 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:42.668969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.668946 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:50:42.684270 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.684220 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" podStartSLOduration=2.301201465 podStartE2EDuration="2.684203781s" totalImagesPullingTime="383.002316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:42.681972576 +0000 UTC m=+1202.285661895" watchObservedRunningTime="2026-10-30 05:50:42.684203781 +0000 UTC m=+1202.287893040" Oct 30 05:50:42.718593 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.718520 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:42.719705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.719678 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:42.720323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:42.720301 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:50:45.617600 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:45.617258 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:50:48.636502 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:48.633595 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:49.769700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:50:49.769667 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:51:00.711513 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:51:00.711408 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:51:05.319392 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.319359 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm"] Oct 30 05:51:05.329037 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.329013 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.346426 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.346396 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm"] Oct 30 05:51:05.362593 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.353504 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-19e33488-serving-cert\"" Oct 30 05:51:05.481508 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.481450 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-openshift-service-ca\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.481711 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.481688 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-metrics-certs\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.482024 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.481998 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7px8t\" (UniqueName: \"kubernetes.io/projected/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-kube-api-access-7px8t\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.482124 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.482109 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.597322 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.584760 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7px8t\" (UniqueName: \"kubernetes.io/projected/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-kube-api-access-7px8t\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.597779 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.597733 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.598034 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.598011 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-openshift-service-ca\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.609760 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.602761 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-metrics-certs\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.609760 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:51:05.604142 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-serving-cert: secret "maas-api-e2e-ait-19e33488-serving-cert" not found Oct 30 05:51:05.609760 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:51:05.604239 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls podName:2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71 nodeName:}" failed. No retries permitted until 2026-10-30 05:51:06.104215622 +0000 UTC m=+1225.707904861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls") pod "maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" (UID: "2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71") : secret "maas-api-e2e-ait-19e33488-serving-cert" not found Oct 30 05:51:05.609760 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.605301 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-openshift-service-ca\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.612277 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.612208 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-metrics-certs\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:05.711935 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:05.711826 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7px8t\" (UniqueName: \"kubernetes.io/projected/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-kube-api-access-7px8t\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:06.110640 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:06.110591 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:06.113653 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:06.113622 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls\") pod \"maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:06.268293 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:06.268251 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:06.577674 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:06.577647 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm"] Oct 30 05:51:06.578240 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:51:06.578210 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5ae057_6afe_4ebf_8f87_f6a4b7ccda71.slice/crio-06cf8cbc9618823d2d57d5cd719f93f85c3c94694c03fa8f14b1b9e8c4468aaa WatchSource:0}: Error finding container 06cf8cbc9618823d2d57d5cd719f93f85c3c94694c03fa8f14b1b9e8c4468aaa: Status 404 returned error can't find the container with id 06cf8cbc9618823d2d57d5cd719f93f85c3c94694c03fa8f14b1b9e8c4468aaa Oct 30 05:51:07.485970 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:07.485933 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" event={"ID":"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71","Type":"ContainerStarted","Data":"fe0ce0cd760bb11dc9d91a59da14b3a513b57acaad88bb87d2a2d137d1b5288f"} Oct 30 05:51:07.486153 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:07.485994 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" event={"ID":"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71","Type":"ContainerStarted","Data":"06cf8cbc9618823d2d57d5cd719f93f85c3c94694c03fa8f14b1b9e8c4468aaa"} Oct 30 05:51:07.486153 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:07.486066 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:07.529801 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:07.529741 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" podStartSLOduration=2.132754171 podStartE2EDuration="2.529722239s" totalImagesPullingTime="396.968068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:07.527348098 +0000 UTC m=+1227.131037375" watchObservedRunningTime="2026-10-30 05:51:07.529722239 +0000 UTC m=+1227.133411497" Oct 30 05:51:13.582486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:13.571824 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:14.685484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:14.684585 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:14.712330 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:14.712282 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:15.213530 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:15.200898 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:15.213530 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:15.202219 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:51:15.708489 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:15.700394 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:51:15.708489 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:15.701265 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:51:16.204485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:16.201225 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:16.204485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:16.202022 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:51:16.229487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:16.221273 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:51:16.229487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:16.221781 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:51:16.757486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:16.751841 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:17.235003 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:17.202256 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:17.235240 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:17.224575 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:17.761136 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:17.758565 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:18.079535 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:18.046019 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Oct 30 05:51:18.274126 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:18.274080 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:18.277987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:18.277936 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:18.398101 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:18.398051 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k"] Oct 30 05:51:18.399443 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:18.399154 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" podUID="0956524a-69aa-4f07-bf32-6511f39615fd" containerName="maas-api" containerID="cri-o://a9f11733051eb47820cd1ebb062930bc8a76b8830d6359d91ae6c8c84e29fa8f" gracePeriod=30 Oct 30 05:51:18.761492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:18.760576 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:51:19.167527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.155032 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4"] Oct 30 05:51:19.327537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.298715 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:19.430532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.366699 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:51:19.479659 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.479627 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4"] Oct 30 05:51:19.480025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.480006 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.741881 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.741842 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.742210 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.742184 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.742382 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.742364 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.742664 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.742639 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7710b253-73f0-4ee8-863a-39e6b5e472a1-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.742872 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.742844 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.743016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.743000 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.743225 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.743207 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.747480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.743434 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.747729 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.747706 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frfx4\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-kube-api-access-frfx4\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.809905 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.809860 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.896725 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.896883 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7710b253-73f0-4ee8-863a-39e6b5e472a1-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.896959 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.897013 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.897125 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.897242 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.897314 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frfx4\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-kube-api-access-frfx4\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.897424 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.897539 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.898121 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.898415 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.974578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.928497 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7710b253-73f0-4ee8-863a-39e6b5e472a1-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:19.976942 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.976910 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb"] Oct 30 05:51:19.994570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:19.994069 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" podUID="cbbeb647-4db1-47e0-8abb-88619bb76907" containerName="istio-proxy" containerID="cri-o://ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211" gracePeriod=30 Oct 30 05:51:20.056655 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.018772 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:20.056655 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.041876 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:20.056655 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.045572 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:20.099488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.097114 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" event={"ID":"0956524a-69aa-4f07-bf32-6511f39615fd","Type":"ContainerDied","Data":"a9f11733051eb47820cd1ebb062930bc8a76b8830d6359d91ae6c8c84e29fa8f"} Oct 30 05:51:20.099488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.097324 5072 generic.go:356] "Generic (PLEG): container finished" podID="0956524a-69aa-4f07-bf32-6511f39615fd" containerID="a9f11733051eb47820cd1ebb062930bc8a76b8830d6359d91ae6c8c84e29fa8f" exitCode=0 Oct 30 05:51:20.110196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.105159 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:51:20.184483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.155245 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:20.231918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.231677 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frfx4\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-kube-api-access-frfx4\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:20.243722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.243687 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:20.297532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.275602 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0956524a-69aa-4f07-bf32-6511f39615fd-kube-api-access-gmqjk\" (UniqueName: \"kubernetes.io/projected/0956524a-69aa-4f07-bf32-6511f39615fd-kube-api-access-gmqjk\") pod \"0956524a-69aa-4f07-bf32-6511f39615fd\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " Oct 30 05:51:20.297532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.275667 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls\") pod \"0956524a-69aa-4f07-bf32-6511f39615fd\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " Oct 30 05:51:20.297532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.275758 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-metrics-certs\") pod \"0956524a-69aa-4f07-bf32-6511f39615fd\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " Oct 30 05:51:20.297532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.276007 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0956524a-69aa-4f07-bf32-6511f39615fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0956524a-69aa-4f07-bf32-6511f39615fd-openshift-service-ca\") pod \"0956524a-69aa-4f07-bf32-6511f39615fd\" (UID: \"0956524a-69aa-4f07-bf32-6511f39615fd\") " Oct 30 05:51:20.327487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.306773 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:51:20.328533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.308662 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0956524a-69aa-4f07-bf32-6511f39615fd-openshift-service-ca" pod "0956524a-69aa-4f07-bf32-6511f39615fd" (UID: "0956524a-69aa-4f07-bf32-6511f39615fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:20.333802 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.333731 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls" pod "0956524a-69aa-4f07-bf32-6511f39615fd" (UID: "0956524a-69aa-4f07-bf32-6511f39615fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:20.342852 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.342818 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-metrics-certs" pod "0956524a-69aa-4f07-bf32-6511f39615fd" (UID: "0956524a-69aa-4f07-bf32-6511f39615fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:20.419627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.370996 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:20.447709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.396797 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:20.502094 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.399816 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:51:20.502094 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.404910 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:20.502094 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.420040 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0956524a-69aa-4f07-bf32-6511f39615fd-kube-api-access-gmqjk" pod "0956524a-69aa-4f07-bf32-6511f39615fd" (UID: "0956524a-69aa-4f07-bf32-6511f39615fd"). InnerVolumeSpecName "kube-api-access-gmqjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:20.503664 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.503643 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0956524a-69aa-4f07-bf32-6511f39615fd-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:20.503856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.503840 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0956524a-69aa-4f07-bf32-6511f39615fd-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:20.628527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.626239 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmqjk\" (UniqueName: \"kubernetes.io/projected/0956524a-69aa-4f07-bf32-6511f39615fd-kube-api-access-gmqjk\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:20.826551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.822881 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:51:20.826551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.822944 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:51:20.844536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.839874 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq"] Oct 30 05:51:20.844536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.843057 5072 cpu_manager.go:414] "containerMap: removing container" podUID="0956524a-69aa-4f07-bf32-6511f39615fd" containerName="maas-api" Oct 30 05:51:20.844536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.843081 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0956524a-69aa-4f07-bf32-6511f39615fd" containerName="maas-api" Oct 30 05:51:20.844536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.843275 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0956524a-69aa-4f07-bf32-6511f39615fd" containerName="maas-api" Oct 30 05:51:20.979171 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.896791 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:20.979171 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.933937 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3224b039-openshift-default-dockercfg-4wklf\"" Oct 30 05:51:21.016613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.987988 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istiod-ca-cert\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.016613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.988120 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nqk\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-kube-api-access-l7nqk\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.016613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.988274 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-certs\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.016613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.988321 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.016613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.988381 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-credential-socket\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.016613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.988413 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-data\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.016613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.988571 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.016613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.988713 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-token\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.016613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:20.988871 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-envoy\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.107854 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.107582 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq"] Oct 30 05:51:21.322615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.296900 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-envoy\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.322615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.297011 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istiod-ca-cert\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.322615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.297111 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nqk\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-kube-api-access-l7nqk\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.322615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.297271 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-certs\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.322615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.297328 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.322615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.297371 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-credential-socket\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.322615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.297405 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-data\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.428028 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.395832 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-data\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.443533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.441199 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.443533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.441325 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-token\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.443533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.442300 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.603512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.589934 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-credential-socket\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.616121 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.601386 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istiod-ca-cert\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.619526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.619495 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-certs\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.778820 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.778723 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" event={"ID":"0956524a-69aa-4f07-bf32-6511f39615fd","Type":"ContainerDied","Data":"638738fb1f2288062b01941f673b4def890ed9865950a338d968b7ed0560242e"} Oct 30 05:51:21.780821 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.780750 5072 scope.go:122] "RemoveContainer" containerID="a9f11733051eb47820cd1ebb062930bc8a76b8830d6359d91ae6c8c84e29fa8f" Oct 30 05:51:21.781234 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.781212 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k" Oct 30 05:51:21.811991 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.810015 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-token\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.881411 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.881330 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.894116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.891887 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-envoy\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.896654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.895882 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nqk\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-kube-api-access-l7nqk\") pod \"e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:21.973189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.947481 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:51:21.973189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.947583 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:51:21.973189 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.947625 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:51:22.000044 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.994565 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4"] Oct 30 05:51:22.000044 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:21.997306 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k"] Oct 30 05:51:22.004502 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:22.004315 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a881dd9-8fbf4f65f-8hp7k"] Oct 30 05:51:22.065120 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:22.065034 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:22.179522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:22.178715 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k"] Oct 30 05:51:22.179522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:22.178955 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" podUID="1951d6e6-b2c1-426d-8b70-2128fb7407a8" containerName="maas-api" containerID="cri-o://6320071def8eefa9a1071f5ee7f52812884205924552b8ffd8df0bf58a0bf47b" gracePeriod=30 Oct 30 05:51:22.903348 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:22.903314 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0956524a-69aa-4f07-bf32-6511f39615fd" path="/var/lib/kubelet/pods/0956524a-69aa-4f07-bf32-6511f39615fd/volumes" Oct 30 05:51:22.913492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:22.909576 5072 generic.go:356] "Generic (PLEG): container finished" podID="1951d6e6-b2c1-426d-8b70-2128fb7407a8" containerID="6320071def8eefa9a1071f5ee7f52812884205924552b8ffd8df0bf58a0bf47b" exitCode=0 Oct 30 05:51:22.913492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:22.909830 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" event={"ID":"1951d6e6-b2c1-426d-8b70-2128fb7407a8","Type":"ContainerDied","Data":"6320071def8eefa9a1071f5ee7f52812884205924552b8ffd8df0bf58a0bf47b"} Oct 30 05:51:22.931487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:22.914243 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" event={"ID":"7710b253-73f0-4ee8-863a-39e6b5e472a1","Type":"ContainerStarted","Data":"f0d0b6ee3135c55178d0ee87aa9dfbe5dab4e710f2e7e9c45620f9232a1048b3"} Oct 30 05:51:23.040485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:23.039667 5072 patch_prober.go:29] interesting pod/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:51:23.040485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:23.039725 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" podUID="cbbeb647-4db1-47e0-8abb-88619bb76907" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:51:23.094787 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:23.094578 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq"] Oct 30 05:51:23.097072 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:51:23.097038 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eacbef0_b3e6_419d_83a4_64ddd49f1d93.slice/crio-b2971d383e70e5731f2e47518604c6178f55712594acca130bbb1c27d76914ac WatchSource:0}: Error finding container b2971d383e70e5731f2e47518604c6178f55712594acca130bbb1c27d76914ac: Status 404 returned error can't find the container with id b2971d383e70e5731f2e47518604c6178f55712594acca130bbb1c27d76914ac Oct 30 05:51:23.100431 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:23.100377 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:51:23.100773 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:23.100751 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:51:23.101039 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:23.101009 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:51:24.064240 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.064119 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5"] Oct 30 05:51:24.095607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.095566 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" podUID="ceae030c-e547-4310-95b5-5db3086fed24" containerName="istio-proxy" containerID="cri-o://f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e" gracePeriod=30 Oct 30 05:51:24.204298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.204261 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" event={"ID":"7710b253-73f0-4ee8-863a-39e6b5e472a1","Type":"ContainerStarted","Data":"f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468"} Oct 30 05:51:24.264198 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.264069 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" event={"ID":"8eacbef0-b3e6-419d-83a4-64ddd49f1d93","Type":"ContainerStarted","Data":"b2971d383e70e5731f2e47518604c6178f55712594acca130bbb1c27d76914ac"} Oct 30 05:51:24.302880 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.302851 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:51:24.305059 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.305014 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podStartSLOduration=6.304998613 podStartE2EDuration="6.304998613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:24.254983856 +0000 UTC m=+1243.858673134" watchObservedRunningTime="2026-10-30 05:51:24.304998613 +0000 UTC m=+1243.908687872" Oct 30 05:51:24.356311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.356272 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls\") pod \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " Oct 30 05:51:24.411485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.402142 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1951d6e6-b2c1-426d-8b70-2128fb7407a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1951d6e6-b2c1-426d-8b70-2128fb7407a8-openshift-service-ca\") pod \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " Oct 30 05:51:24.438616 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.438576 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:24.462273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.460100 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1951d6e6-b2c1-426d-8b70-2128fb7407a8-kube-api-access-7wzgg\" (UniqueName: \"kubernetes.io/projected/1951d6e6-b2c1-426d-8b70-2128fb7407a8-kube-api-access-7wzgg\") pod \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " Oct 30 05:51:24.462273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.460248 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-metrics-certs\") pod \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\" (UID: \"1951d6e6-b2c1-426d-8b70-2128fb7407a8\") " Oct 30 05:51:24.462273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.460409 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1951d6e6-b2c1-426d-8b70-2128fb7407a8-openshift-service-ca" pod "1951d6e6-b2c1-426d-8b70-2128fb7407a8" (UID: "1951d6e6-b2c1-426d-8b70-2128fb7407a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:24.462273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.461210 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1951d6e6-b2c1-426d-8b70-2128fb7407a8-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:24.487351 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.487309 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:51:24.487630 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.487573 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 05:51:24.692981 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.692944 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls" pod "1951d6e6-b2c1-426d-8b70-2128fb7407a8" (UID: "1951d6e6-b2c1-426d-8b70-2128fb7407a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:24.695427 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.695372 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-metrics-certs" pod "1951d6e6-b2c1-426d-8b70-2128fb7407a8" (UID: "1951d6e6-b2c1-426d-8b70-2128fb7407a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:24.695427 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.695377 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1951d6e6-b2c1-426d-8b70-2128fb7407a8-kube-api-access-7wzgg" pod "1951d6e6-b2c1-426d-8b70-2128fb7407a8" (UID: "1951d6e6-b2c1-426d-8b70-2128fb7407a8"). InnerVolumeSpecName "kube-api-access-7wzgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:24.797482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.793289 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wzgg\" (UniqueName: \"kubernetes.io/projected/1951d6e6-b2c1-426d-8b70-2128fb7407a8-kube-api-access-7wzgg\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:24.797482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.793322 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:24.797482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:24.793335 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1951d6e6-b2c1-426d-8b70-2128fb7407a8-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:25.273696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:25.273659 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" event={"ID":"8eacbef0-b3e6-419d-83a4-64ddd49f1d93","Type":"ContainerStarted","Data":"c8cc167164c721b991effc2f03897ae2be0f0c4cf72abe2e509441c48ee49be4"} Oct 30 05:51:25.275260 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:25.275228 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" event={"ID":"1951d6e6-b2c1-426d-8b70-2128fb7407a8","Type":"ContainerDied","Data":"185f8912e20f322ee9cbe67a142a50905c65b1c2d2b6b9429d9c911b25d36af0"} Oct 30 05:51:25.275402 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:25.275258 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k" Oct 30 05:51:25.275402 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:25.275294 5072 scope.go:122] "RemoveContainer" containerID="6320071def8eefa9a1071f5ee7f52812884205924552b8ffd8df0bf58a0bf47b" Oct 30 05:51:25.385675 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:25.385605 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" podStartSLOduration=5.38558569 podStartE2EDuration="5.38558569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:25.342924215 +0000 UTC m=+1244.946613535" watchObservedRunningTime="2026-10-30 05:51:25.38558569 +0000 UTC m=+1244.989274954" Oct 30 05:51:25.386176 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:25.386154 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k"] Oct 30 05:51:25.390888 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:25.390862 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3cc280cf-7dd54d58b7-s222k"] Oct 30 05:51:25.397771 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:25.397707 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:51:25.397771 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:25.397758 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 05:51:26.065434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:26.065400 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:26.074106 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:26.073806 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:26.284842 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:26.284811 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:26.286030 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:26.286006 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:51:26.399486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:26.397593 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:51:26.399486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:26.397649 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 05:51:26.563803 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:26.563762 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1951d6e6-b2c1-426d-8b70-2128fb7407a8" path="/var/lib/kubelet/pods/1951d6e6-b2c1-426d-8b70-2128fb7407a8/volumes" Oct 30 05:51:27.398099 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:27.398071 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:51:27.398635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:27.398599 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 05:51:28.397268 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:28.397223 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:51:28.397503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:28.397287 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 05:51:29.472627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:29.464062 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:51:29.472627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:29.464164 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 05:51:30.457111 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:30.457076 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:51:30.457404 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:30.457376 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 05:51:30.726065 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:51:30.725983 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:51:31.398074 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:31.398040 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:51:31.398531 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:31.398491 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 05:51:32.403257 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:32.403219 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:32.403803 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:32.403702 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:32.404594 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:32.404571 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 05:51:33.009332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.009304 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:51:33.194383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.194278 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-credential-socket\") pod \"cbbeb647-4db1-47e0-8abb-88619bb76907\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " Oct 30 05:51:33.194383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.194374 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-token\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-token\") pod \"cbbeb647-4db1-47e0-8abb-88619bb76907\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " Oct 30 05:51:33.194608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.194506 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-data\") pod \"cbbeb647-4db1-47e0-8abb-88619bb76907\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " Oct 30 05:51:33.194608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.194550 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-podinfo\") pod \"cbbeb647-4db1-47e0-8abb-88619bb76907\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " Oct 30 05:51:33.194703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.194684 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-socket\") pod \"cbbeb647-4db1-47e0-8abb-88619bb76907\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " Oct 30 05:51:33.194794 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.194771 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-certs\") pod \"cbbeb647-4db1-47e0-8abb-88619bb76907\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " Oct 30 05:51:33.194898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.194814 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbbeb647-4db1-47e0-8abb-88619bb76907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbeb647-4db1-47e0-8abb-88619bb76907-istiod-ca-cert\") pod \"cbbeb647-4db1-47e0-8abb-88619bb76907\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " Oct 30 05:51:33.194898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.194848 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-envoy\") pod \"cbbeb647-4db1-47e0-8abb-88619bb76907\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " Oct 30 05:51:33.195058 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.194964 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-kube-api-access-f9vk6\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-kube-api-access-f9vk6\") pod \"cbbeb647-4db1-47e0-8abb-88619bb76907\" (UID: \"cbbeb647-4db1-47e0-8abb-88619bb76907\") " Oct 30 05:51:33.195243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.195218 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-credential-socket" pod "cbbeb647-4db1-47e0-8abb-88619bb76907" (UID: "cbbeb647-4db1-47e0-8abb-88619bb76907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.195931 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.195829 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-socket" pod "cbbeb647-4db1-47e0-8abb-88619bb76907" (UID: "cbbeb647-4db1-47e0-8abb-88619bb76907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.195931 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.195843 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.196261 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.196134 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbeb647-4db1-47e0-8abb-88619bb76907-istiod-ca-cert" pod "cbbeb647-4db1-47e0-8abb-88619bb76907" (UID: "cbbeb647-4db1-47e0-8abb-88619bb76907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:33.196432 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.196399 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-certs" pod "cbbeb647-4db1-47e0-8abb-88619bb76907" (UID: "cbbeb647-4db1-47e0-8abb-88619bb76907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.196432 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.196424 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-data" pod "cbbeb647-4db1-47e0-8abb-88619bb76907" (UID: "cbbeb647-4db1-47e0-8abb-88619bb76907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.198812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.198777 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-kube-api-access-f9vk6" pod "cbbeb647-4db1-47e0-8abb-88619bb76907" (UID: "cbbeb647-4db1-47e0-8abb-88619bb76907"). InnerVolumeSpecName "kube-api-access-f9vk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:33.199083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.199056 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-podinfo" pod "cbbeb647-4db1-47e0-8abb-88619bb76907" (UID: "cbbeb647-4db1-47e0-8abb-88619bb76907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:51:33.199981 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.199961 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-envoy" pod "cbbeb647-4db1-47e0-8abb-88619bb76907" (UID: "cbbeb647-4db1-47e0-8abb-88619bb76907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.200642 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.200624 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-token" pod "cbbeb647-4db1-47e0-8abb-88619bb76907" (UID: "cbbeb647-4db1-47e0-8abb-88619bb76907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:33.225255 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.225236 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:51:33.297616 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.297583 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.297616 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.297611 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.297616 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.297621 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbeb647-4db1-47e0-8abb-88619bb76907-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.297616 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.297629 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.297988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.297639 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9vk6\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-kube-api-access-f9vk6\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.297988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.297648 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.297988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.297657 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.297988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.297665 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbeb647-4db1-47e0-8abb-88619bb76907-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.398282 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398252 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-kube-api-access-2qfcz\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-kube-api-access-2qfcz\") pod \"ceae030c-e547-4310-95b5-5db3086fed24\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " Oct 30 05:51:33.398451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398290 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-istio-token\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-istio-token\") pod \"ceae030c-e547-4310-95b5-5db3086fed24\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " Oct 30 05:51:33.398451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398421 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-certs\") pod \"ceae030c-e547-4310-95b5-5db3086fed24\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " Oct 30 05:51:33.398592 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398529 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-socket\") pod \"ceae030c-e547-4310-95b5-5db3086fed24\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " Oct 30 05:51:33.398592 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398569 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceae030c-e547-4310-95b5-5db3086fed24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae030c-e547-4310-95b5-5db3086fed24-istiod-ca-cert\") pod \"ceae030c-e547-4310-95b5-5db3086fed24\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " Oct 30 05:51:33.398739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398712 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-certs" pod "ceae030c-e547-4310-95b5-5db3086fed24" (UID: "ceae030c-e547-4310-95b5-5db3086fed24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.398817 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398737 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-envoy\") pod \"ceae030c-e547-4310-95b5-5db3086fed24\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " Oct 30 05:51:33.398817 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398792 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceae030c-e547-4310-95b5-5db3086fed24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae030c-e547-4310-95b5-5db3086fed24-istio-podinfo\") pod \"ceae030c-e547-4310-95b5-5db3086fed24\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " Oct 30 05:51:33.398931 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398828 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-credential-socket\") pod \"ceae030c-e547-4310-95b5-5db3086fed24\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " Oct 30 05:51:33.398931 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398856 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-data\") pod \"ceae030c-e547-4310-95b5-5db3086fed24\" (UID: \"ceae030c-e547-4310-95b5-5db3086fed24\") " Oct 30 05:51:33.399038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398979 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceae030c-e547-4310-95b5-5db3086fed24-istiod-ca-cert" pod "ceae030c-e547-4310-95b5-5db3086fed24" (UID: "ceae030c-e547-4310-95b5-5db3086fed24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:33.399163 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.398813 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-socket" pod "ceae030c-e547-4310-95b5-5db3086fed24" (UID: "ceae030c-e547-4310-95b5-5db3086fed24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.399280 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.399232 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-credential-socket" pod "ceae030c-e547-4310-95b5-5db3086fed24" (UID: "ceae030c-e547-4310-95b5-5db3086fed24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.399343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.399299 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-data" pod "ceae030c-e547-4310-95b5-5db3086fed24" (UID: "ceae030c-e547-4310-95b5-5db3086fed24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.399998 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.399974 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.400105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.400003 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.400105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.400018 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceae030c-e547-4310-95b5-5db3086fed24-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.400105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.400032 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.400105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.400047 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.401120 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.401095 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-envoy" pod "ceae030c-e547-4310-95b5-5db3086fed24" (UID: "ceae030c-e547-4310-95b5-5db3086fed24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:51:33.401120 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.401107 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceae030c-e547-4310-95b5-5db3086fed24-istio-podinfo" pod "ceae030c-e547-4310-95b5-5db3086fed24" (UID: "ceae030c-e547-4310-95b5-5db3086fed24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:51:33.401357 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.401331 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-kube-api-access-2qfcz" pod "ceae030c-e547-4310-95b5-5db3086fed24" (UID: "ceae030c-e547-4310-95b5-5db3086fed24"). InnerVolumeSpecName "kube-api-access-2qfcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:33.401398 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.401361 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-istio-token" pod "ceae030c-e547-4310-95b5-5db3086fed24" (UID: "ceae030c-e547-4310-95b5-5db3086fed24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:33.501608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.501528 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qfcz\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-kube-api-access-2qfcz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.501608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.501566 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceae030c-e547-4310-95b5-5db3086fed24-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.501608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.501582 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceae030c-e547-4310-95b5-5db3086fed24-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.501608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.501593 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceae030c-e547-4310-95b5-5db3086fed24-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.581433 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.581394 5072 generic.go:356] "Generic (PLEG): container finished" podID="ceae030c-e547-4310-95b5-5db3086fed24" containerID="f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e" exitCode=0 Oct 30 05:51:33.581640 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.581446 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" Oct 30 05:51:33.581640 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.581488 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" event={"ID":"ceae030c-e547-4310-95b5-5db3086fed24","Type":"ContainerDied","Data":"f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e"} Oct 30 05:51:33.581640 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.581553 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5" event={"ID":"ceae030c-e547-4310-95b5-5db3086fed24","Type":"ContainerDied","Data":"b771b135eb4722fda0d43b2625355c853d302dca7718a47a32793524866989a1"} Oct 30 05:51:33.581640 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.581585 5072 scope.go:122] "RemoveContainer" containerID="f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e" Oct 30 05:51:33.583179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.583155 5072 generic.go:356] "Generic (PLEG): container finished" podID="cbbeb647-4db1-47e0-8abb-88619bb76907" containerID="ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211" exitCode=0 Oct 30 05:51:33.583447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.583207 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" Oct 30 05:51:33.583447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.583203 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" event={"ID":"cbbeb647-4db1-47e0-8abb-88619bb76907","Type":"ContainerDied","Data":"ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211"} Oct 30 05:51:33.583447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.583334 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb" event={"ID":"cbbeb647-4db1-47e0-8abb-88619bb76907","Type":"ContainerDied","Data":"380b08ba7eb7e4be583061d11627476ebf6c8e43228c4cbed225cd4a1763dbb1"} Oct 30 05:51:33.594385 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.594364 5072 scope.go:122] "RemoveContainer" containerID="f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e" Oct 30 05:51:33.594686 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:51:33.594668 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e\": container with ID starting with f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e not found: ID does not exist" containerID="f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e" Oct 30 05:51:33.594770 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.594697 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e"} err="failed to get container status \"f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e\": rpc error: code = NotFound desc = could not find container \"f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e\": container with ID starting with f89332e9777aa664518c5006ca22bad8a7920a6bd3de4c4c526220fdde420e1e not found: ID does not exist" Oct 30 05:51:33.594770 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.594722 5072 scope.go:122] "RemoveContainer" containerID="ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211" Oct 30 05:51:33.605858 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.605841 5072 scope.go:122] "RemoveContainer" containerID="ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211" Oct 30 05:51:33.606165 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:51:33.606147 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211\": container with ID starting with ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211 not found: ID does not exist" containerID="ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211" Oct 30 05:51:33.606242 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.606183 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211"} err="failed to get container status \"ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211\": rpc error: code = NotFound desc = could not find container \"ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211\": container with ID starting with ab86eea5e1a2ddb528cb3644f9f5ba5f9b731bbb0334a3d812847992dbb17211 not found: ID does not exist" Oct 30 05:51:33.634234 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.634205 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb"] Oct 30 05:51:33.645769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.645741 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6a881dd9-openshift-default-54766b7c9f-7s4cb"] Oct 30 05:51:33.672381 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.672339 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5"] Oct 30 05:51:33.677529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:33.677505 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3cc280cf-gw-openshift-default-54bbcd55c-gkjq5"] Oct 30 05:51:34.522294 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:34.522253 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbbeb647-4db1-47e0-8abb-88619bb76907" path="/var/lib/kubelet/pods/cbbeb647-4db1-47e0-8abb-88619bb76907/volumes" Oct 30 05:51:34.523054 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:34.523032 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceae030c-e547-4310-95b5-5db3086fed24" path="/var/lib/kubelet/pods/ceae030c-e547-4310-95b5-5db3086fed24/volumes" Oct 30 05:51:43.914787 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.914709 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf"] Oct 30 05:51:43.915550 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.915529 5072 cpu_manager.go:414] "containerMap: removing container" podUID="ceae030c-e547-4310-95b5-5db3086fed24" containerName="istio-proxy" Oct 30 05:51:43.915550 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.915548 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceae030c-e547-4310-95b5-5db3086fed24" containerName="istio-proxy" Oct 30 05:51:43.915706 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.915558 5072 cpu_manager.go:414] "containerMap: removing container" podUID="1951d6e6-b2c1-426d-8b70-2128fb7407a8" containerName="maas-api" Oct 30 05:51:43.915706 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.915563 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1951d6e6-b2c1-426d-8b70-2128fb7407a8" containerName="maas-api" Oct 30 05:51:43.915706 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.915582 5072 cpu_manager.go:414] "containerMap: removing container" podUID="cbbeb647-4db1-47e0-8abb-88619bb76907" containerName="istio-proxy" Oct 30 05:51:43.915706 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.915587 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbbeb647-4db1-47e0-8abb-88619bb76907" containerName="istio-proxy" Oct 30 05:51:43.915897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.915716 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceae030c-e547-4310-95b5-5db3086fed24" containerName="istio-proxy" Oct 30 05:51:43.915897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.915725 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1951d6e6-b2c1-426d-8b70-2128fb7407a8" containerName="maas-api" Oct 30 05:51:43.915897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.915736 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbbeb647-4db1-47e0-8abb-88619bb76907" containerName="istio-proxy" Oct 30 05:51:43.919860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.919839 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:43.951179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.951131 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3224b039-serving-cert\"" Oct 30 05:51:43.958756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:43.958731 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf"] Oct 30 05:51:44.091077 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.091039 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-metrics-certs\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.091254 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.091149 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/310fc624-b6d0-4f92-a7d9-f8e54ac29631-kube-api-access-855rh\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.091311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.091286 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-maas-api-tls\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.091549 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.091527 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310fc624-b6d0-4f92-a7d9-f8e54ac29631-openshift-service-ca\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.204850 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.204740 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310fc624-b6d0-4f92-a7d9-f8e54ac29631-openshift-service-ca\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.205514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.205486 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-metrics-certs\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.205734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.205711 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/310fc624-b6d0-4f92-a7d9-f8e54ac29631-kube-api-access-855rh\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.211687 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.211657 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-maas-api-tls\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.221851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.221820 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310fc624-b6d0-4f92-a7d9-f8e54ac29631-openshift-service-ca\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.238485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.232631 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-maas-api-tls\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.265773 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.265737 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/310fc624-b6d0-4f92-a7d9-f8e54ac29631-kube-api-access-855rh\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.282484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.276025 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-metrics-certs\") pod \"maas-api-e2e-derive-3224b039-85f9869478-b65gf\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.537099 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.537008 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:44.866729 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:44.866688 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf"] Oct 30 05:51:44.867377 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:51:44.867351 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310fc624_b6d0_4f92_a7d9_f8e54ac29631.slice/crio-4ff0a5d9cd88a9a6d14f8d96bc45dd9e99b9e9de2daa0d4fccd04fb6530dbe24 WatchSource:0}: Error finding container 4ff0a5d9cd88a9a6d14f8d96bc45dd9e99b9e9de2daa0d4fccd04fb6530dbe24: Status 404 returned error can't find the container with id 4ff0a5d9cd88a9a6d14f8d96bc45dd9e99b9e9de2daa0d4fccd04fb6530dbe24 Oct 30 05:51:45.555513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:45.554575 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm"] Oct 30 05:51:45.555513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:45.554826 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" podUID="2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" containerName="maas-api" containerID="cri-o://fe0ce0cd760bb11dc9d91a59da14b3a513b57acaad88bb87d2a2d137d1b5288f" gracePeriod=30 Oct 30 05:51:46.029512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.029447 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" event={"ID":"310fc624-b6d0-4f92-a7d9-f8e54ac29631","Type":"ContainerStarted","Data":"4ff0a5d9cd88a9a6d14f8d96bc45dd9e99b9e9de2daa0d4fccd04fb6530dbe24"} Oct 30 05:51:46.076398 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.070255 5072 generic.go:356] "Generic (PLEG): container finished" podID="2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" containerID="fe0ce0cd760bb11dc9d91a59da14b3a513b57acaad88bb87d2a2d137d1b5288f" exitCode=0 Oct 30 05:51:46.083247 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.070445 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" event={"ID":"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71","Type":"ContainerDied","Data":"fe0ce0cd760bb11dc9d91a59da14b3a513b57acaad88bb87d2a2d137d1b5288f"} Oct 30 05:51:46.797338 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.797309 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:46.906657 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.906628 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls\") pod \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " Oct 30 05:51:46.906841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.906750 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-kube-api-access-7px8t\" (UniqueName: \"kubernetes.io/projected/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-kube-api-access-7px8t\") pod \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " Oct 30 05:51:46.906841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.906799 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-metrics-certs\") pod \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " Oct 30 05:51:46.906972 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.906910 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-openshift-service-ca\") pod \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\" (UID: \"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71\") " Oct 30 05:51:46.907324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.907295 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-openshift-service-ca" pod "2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" (UID: "2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:46.907626 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.907606 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:46.909132 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.909100 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls" pod "2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" (UID: "2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:46.909580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.909555 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-metrics-certs" pod "2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" (UID: "2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:46.909788 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:46.909769 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-kube-api-access-7px8t" pod "2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" (UID: "2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71"). InnerVolumeSpecName "kube-api-access-7px8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:47.009337 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.009304 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:47.009337 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.009338 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7px8t\" (UniqueName: \"kubernetes.io/projected/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-kube-api-access-7px8t\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:47.009567 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.009354 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:51:47.104978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.104356 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" event={"ID":"310fc624-b6d0-4f92-a7d9-f8e54ac29631","Type":"ContainerStarted","Data":"6c1ee046f1f20eee2e7eb7af899f4144400660a2fd05eeca0d13a5d1587ce5ab"} Oct 30 05:51:47.104978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.104737 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:47.152303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.152264 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" event={"ID":"2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71","Type":"ContainerDied","Data":"06cf8cbc9618823d2d57d5cd719f93f85c3c94694c03fa8f14b1b9e8c4468aaa"} Oct 30 05:51:47.180534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.176609 5072 scope.go:122] "RemoveContainer" containerID="fe0ce0cd760bb11dc9d91a59da14b3a513b57acaad88bb87d2a2d137d1b5288f" Oct 30 05:51:47.180534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.152563 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm" Oct 30 05:51:47.303719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.283533 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" podStartSLOduration=3.6131327989999997 podStartE2EDuration="4.283510976s" totalImagesPullingTime="670.378177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:47.191756731 +0000 UTC m=+1266.795445989" watchObservedRunningTime="2026-10-30 05:51:47.283510976 +0000 UTC m=+1266.887200253" Oct 30 05:51:47.318747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.318681 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm"] Oct 30 05:51:47.379788 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:47.370384 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19e33488-7b56b4d949-w5xbm"] Oct 30 05:51:48.531521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:48.531479 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" path="/var/lib/kubelet/pods/2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71/volumes" Oct 30 05:51:52.102765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.102729 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b"] Oct 30 05:51:52.104960 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.104900 5072 cpu_manager.go:414] "containerMap: removing container" podUID="2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" containerName="maas-api" Oct 30 05:51:52.105251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.105231 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" containerName="maas-api" Oct 30 05:51:52.105682 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.105663 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5ae057-6afe-4ebf-8f87-f6a4b7ccda71" containerName="maas-api" Oct 30 05:51:52.115344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.115322 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.134536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.134501 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b"] Oct 30 05:51:52.136873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.136849 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"kube-root-ca.crt\"" Oct 30 05:51:52.142967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.142941 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"openshift-service-ca.crt\"" Oct 30 05:51:52.142967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.142962 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"distinct-w0-a14091-kserve-self-signed-certs\"" Oct 30 05:51:52.143177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.143151 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"default-dockercfg-c4xwk\"" Oct 30 05:51:52.192921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.192821 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tmp-dir\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.193803 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.193777 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tls-certs\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.193934 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.193894 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-home\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.194003 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.193988 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-model-cache\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.194068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.194053 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-dshm\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.194246 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.194223 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxx26\" (UniqueName: \"kubernetes.io/projected/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-kube-api-access-qxx26\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.296944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.296602 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tmp-dir\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.296944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.296677 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tls-certs\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.296944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.296782 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-home\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.296944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.296897 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-model-cache\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.297308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.296991 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-dshm\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.297308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.297149 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxx26\" (UniqueName: \"kubernetes.io/projected/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-kube-api-access-qxx26\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.300527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.298223 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tmp-dir\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.300527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.299045 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-model-cache\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.300527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.299280 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-home\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.309483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.303208 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tls-certs\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.336489 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.335147 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-dshm\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.355284 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.349376 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxx26\" (UniqueName: \"kubernetes.io/projected/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-kube-api-access-qxx26\") pod \"distinct-w0-a14091-kserve-56c886dd5c-hmn2b\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.444494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.440843 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:51:52.874316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:52.874284 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b"] Oct 30 05:51:52.875257 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:51:52.875184 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaa69bc4_be2f_4acc_a8a2_4ba5ae42a507.slice/crio-15b0ac7a7991ade895c5729a868c91c50de5cdb6a1e9b9e915d6e4f3f93fc146 WatchSource:0}: Error finding container 15b0ac7a7991ade895c5729a868c91c50de5cdb6a1e9b9e915d6e4f3f93fc146: Status 404 returned error can't find the container with id 15b0ac7a7991ade895c5729a868c91c50de5cdb6a1e9b9e915d6e4f3f93fc146 Oct 30 05:51:53.168352 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:53.168262 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:51:53.254039 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:53.254000 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" event={"ID":"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507","Type":"ContainerStarted","Data":"00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490"} Oct 30 05:51:53.254231 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:53.254055 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" event={"ID":"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507","Type":"ContainerStarted","Data":"15b0ac7a7991ade895c5729a868c91c50de5cdb6a1e9b9e915d6e4f3f93fc146"} Oct 30 05:51:53.304326 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:53.304262 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podStartSLOduration=1.304241782 podStartE2EDuration="1.304241782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:53.302594141 +0000 UTC m=+1272.906283422" watchObservedRunningTime="2026-10-30 05:51:53.304241782 +0000 UTC m=+1272.907931041" Oct 30 05:51:53.449610 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:51:53.448504 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:52:00.711434 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:52:00.711395 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:52:02.441776 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:02.441741 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:52:02.455262 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:02.455229 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:52:03.385951 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:03.385921 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:52:15.490392 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.490354 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh"] Oct 30 05:52:15.501214 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.500171 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.523888 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.523856 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh"] Oct 30 05:52:15.532172 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.531344 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-0b481b-openshift-default-dockercfg-nrg7r\"" Oct 30 05:52:15.782538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.767845 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.782538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.767953 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.782538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.768008 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.782538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.768082 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.782538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.768122 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.782538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.768248 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45652c65-84b3-4463-92fe-9bb9a8e230f7-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.782538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.768375 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.782538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.768543 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.782538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.768571 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2td2w\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-kube-api-access-2td2w\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.897363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.885257 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.897363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.885312 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2td2w\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-kube-api-access-2td2w\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.897363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.885412 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.897363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.890194 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.901023 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.900979 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.901217 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.901054 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.901217 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.901137 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.901217 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.901162 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.901396 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.901282 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45652c65-84b3-4463-92fe-9bb9a8e230f7-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.901396 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.901385 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.901819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.901790 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.903378 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.902582 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45652c65-84b3-4463-92fe-9bb9a8e230f7-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.903378 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.903173 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.905148 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.903696 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.907847 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.907799 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.948500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.945919 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.967308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.967242 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2td2w\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-kube-api-access-2td2w\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:15.967909 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:15.967869 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:16.135641 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:16.135602 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:16.508162 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:16.508056 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf"] Oct 30 05:52:16.534432 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:16.508340 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" podUID="310fc624-b6d0-4f92-a7d9-f8e54ac29631" containerName="maas-api" containerID="cri-o://6c1ee046f1f20eee2e7eb7af899f4144400660a2fd05eeca0d13a5d1587ce5ab" gracePeriod=30 Oct 30 05:52:16.563067 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:16.563003 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh"] Oct 30 05:52:16.585812 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:52:16.585770 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45652c65_84b3_4463_92fe_9bb9a8e230f7.slice/crio-d711cc4ca81b6a4c465f8968679aa6831670539de31f905c57853174f220abee WatchSource:0}: Error finding container d711cc4ca81b6a4c465f8968679aa6831670539de31f905c57853174f220abee: Status 404 returned error can't find the container with id d711cc4ca81b6a4c465f8968679aa6831670539de31f905c57853174f220abee Oct 30 05:52:16.633050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:16.632995 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:16.633195 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:16.633093 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:16.633195 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:16.633141 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:16.953203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:16.953090 5072 generic.go:356] "Generic (PLEG): container finished" podID="310fc624-b6d0-4f92-a7d9-f8e54ac29631" containerID="6c1ee046f1f20eee2e7eb7af899f4144400660a2fd05eeca0d13a5d1587ce5ab" exitCode=0 Oct 30 05:52:16.953682 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:16.953217 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" event={"ID":"310fc624-b6d0-4f92-a7d9-f8e54ac29631","Type":"ContainerDied","Data":"6c1ee046f1f20eee2e7eb7af899f4144400660a2fd05eeca0d13a5d1587ce5ab"} Oct 30 05:52:17.145521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.123624 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" event={"ID":"45652c65-84b3-4463-92fe-9bb9a8e230f7","Type":"ContainerStarted","Data":"fcb861e348d08f9bc45bacd3b6e51b0a9388bb2916acf9d6de2440c7e4b5b185"} Oct 30 05:52:17.145521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.123701 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" event={"ID":"45652c65-84b3-4463-92fe-9bb9a8e230f7","Type":"ContainerStarted","Data":"d711cc4ca81b6a4c465f8968679aa6831670539de31f905c57853174f220abee"} Oct 30 05:52:17.150061 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.149964 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:17.194355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.178364 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:52:17.213833 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.213770 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:52:17.223777 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.223146 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:52:17.358294 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.332261 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310fc624-b6d0-4f92-a7d9-f8e54ac29631-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310fc624-b6d0-4f92-a7d9-f8e54ac29631-openshift-service-ca\") pod \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " Oct 30 05:52:17.358294 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.332411 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310fc624-b6d0-4f92-a7d9-f8e54ac29631-kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/310fc624-b6d0-4f92-a7d9-f8e54ac29631-kube-api-access-855rh\") pod \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " Oct 30 05:52:17.358294 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.332580 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-metrics-certs\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-metrics-certs\") pod \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " Oct 30 05:52:17.358294 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.332662 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-maas-api-tls\") pod \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\" (UID: \"310fc624-b6d0-4f92-a7d9-f8e54ac29631\") " Oct 30 05:52:17.361058 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.360974 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310fc624-b6d0-4f92-a7d9-f8e54ac29631-openshift-service-ca" pod "310fc624-b6d0-4f92-a7d9-f8e54ac29631" (UID: "310fc624-b6d0-4f92-a7d9-f8e54ac29631"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:17.383483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.370210 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podStartSLOduration=2.37018851 podStartE2EDuration="2.37018851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:17.265515004 +0000 UTC m=+1296.869204275" watchObservedRunningTime="2026-10-30 05:52:17.37018851 +0000 UTC m=+1296.973877766" Oct 30 05:52:17.436302 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.434661 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310fc624-b6d0-4f92-a7d9-f8e54ac29631-kube-api-access-855rh" pod "310fc624-b6d0-4f92-a7d9-f8e54ac29631" (UID: "310fc624-b6d0-4f92-a7d9-f8e54ac29631"). InnerVolumeSpecName "kube-api-access-855rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:17.445542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.437433 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855rh\" (UniqueName: \"kubernetes.io/projected/310fc624-b6d0-4f92-a7d9-f8e54ac29631-kube-api-access-855rh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:17.445861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.445783 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/310fc624-b6d0-4f92-a7d9-f8e54ac29631-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:17.445861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.442383 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-metrics-certs" pod "310fc624-b6d0-4f92-a7d9-f8e54ac29631" (UID: "310fc624-b6d0-4f92-a7d9-f8e54ac29631"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:17.445861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.442597 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-maas-api-tls" pod "310fc624-b6d0-4f92-a7d9-f8e54ac29631" (UID: "310fc624-b6d0-4f92-a7d9-f8e54ac29631"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:17.590058 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.586012 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:17.590058 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.586046 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/310fc624-b6d0-4f92-a7d9-f8e54ac29631-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:17.928481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.927257 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq"] Oct 30 05:52:17.928481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:17.927534 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" podUID="8eacbef0-b3e6-419d-83a4-64ddd49f1d93" containerName="istio-proxy" containerID="cri-o://c8cc167164c721b991effc2f03897ae2be0f0c4cf72abe2e509441c48ee49be4" gracePeriod=30 Oct 30 05:52:18.242537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:18.220899 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:18.352484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:18.317303 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" Oct 30 05:52:18.426806 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:18.323390 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf" event={"ID":"310fc624-b6d0-4f92-a7d9-f8e54ac29631","Type":"ContainerDied","Data":"4ff0a5d9cd88a9a6d14f8d96bc45dd9e99b9e9de2daa0d4fccd04fb6530dbe24"} Oct 30 05:52:18.426806 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:18.415920 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:18.426806 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:18.416075 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:52:18.426806 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:18.416119 5072 scope.go:122] "RemoveContainer" containerID="6c1ee046f1f20eee2e7eb7af899f4144400660a2fd05eeca0d13a5d1587ce5ab" Oct 30 05:52:18.643317 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:18.642937 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf"] Oct 30 05:52:18.650810 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:18.650780 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3224b039-85f9869478-b65gf"] Oct 30 05:52:19.466517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.466478 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs"] Oct 30 05:52:19.467729 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.467706 5072 cpu_manager.go:414] "containerMap: removing container" podUID="310fc624-b6d0-4f92-a7d9-f8e54ac29631" containerName="maas-api" Oct 30 05:52:19.467804 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.467732 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310fc624-b6d0-4f92-a7d9-f8e54ac29631" containerName="maas-api" Oct 30 05:52:19.468023 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.468007 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="310fc624-b6d0-4f92-a7d9-f8e54ac29631" containerName="maas-api" Oct 30 05:52:19.474206 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.474184 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.493071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.493041 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs"] Oct 30 05:52:19.499856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.499827 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8bab43d7-openshift-default-dockercfg-62h4r\"" Oct 30 05:52:19.637687 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.637643 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpn97\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-kube-api-access-rpn97\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.638026 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.638002 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-token\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.642699 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.642664 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-socket\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.725551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.716580 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-envoy\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.725551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.716670 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-certs\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.725551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.716758 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-podinfo\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.725551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.716795 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-data\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.725551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.716864 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istiod-ca-cert\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.725551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.716926 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-credential-socket\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.760484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.753870 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm"] Oct 30 05:52:19.760484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.754131 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" containerID="cri-o://1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a" gracePeriod=30 Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.828582 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-socket\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.828636 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-envoy\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.828699 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-certs\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.828775 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-podinfo\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.828809 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-data\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.828865 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istiod-ca-cert\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.828914 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-credential-socket\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.829024 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpn97\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-kube-api-access-rpn97\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.829142 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-token\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.830482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.829979 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-socket\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.907309 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.860994 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-data\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.907309 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.861238 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-certs\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.931524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.909897 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istiod-ca-cert\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:19.956373 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:19.956339 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-credential-socket\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:20.136852 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:20.136814 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-podinfo\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:20.137238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:20.137211 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-token\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:20.137533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:20.137502 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-envoy\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:20.140834 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:20.140808 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpn97\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-kube-api-access-rpn97\") pod \"e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:20.143615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:20.142904 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv"] Oct 30 05:52:20.143615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:20.143159 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" containerID="cri-o://e82d050862f95b8bd0df1612d8b818e04bb147f679b32b2269193ad700d98bca" gracePeriod=30 Oct 30 05:52:20.438937 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:20.438837 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:20.669653 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:20.669620 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310fc624-b6d0-4f92-a7d9-f8e54ac29631" path="/var/lib/kubelet/pods/310fc624-b6d0-4f92-a7d9-f8e54ac29631/volumes" Oct 30 05:52:21.022632 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:21.022602 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs"] Oct 30 05:52:21.026072 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:21.026040 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:21.026190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:21.026120 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:21.026190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:21.026162 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:21.451384 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:21.449852 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" event={"ID":"cf2b7891-fa63-4011-beaa-aa8692e24cd7","Type":"ContainerStarted","Data":"d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777"} Oct 30 05:52:21.451384 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:21.449939 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" event={"ID":"cf2b7891-fa63-4011-beaa-aa8692e24cd7","Type":"ContainerStarted","Data":"678d1aa1be19652b9327fba7a1b71f6fed96bee76c838a0bda80bbb01614bf00"} Oct 30 05:52:21.531381 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:21.531322 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podStartSLOduration=2.53130056 podStartE2EDuration="2.53130056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:21.516696245 +0000 UTC m=+1301.120385503" watchObservedRunningTime="2026-10-30 05:52:21.53130056 +0000 UTC m=+1301.134989819" Oct 30 05:52:22.446600 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.446565 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:22.464147 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.464065 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:22.466262 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.466222 5072 generic.go:356] "Generic (PLEG): container finished" podID="8eacbef0-b3e6-419d-83a4-64ddd49f1d93" containerID="c8cc167164c721b991effc2f03897ae2be0f0c4cf72abe2e509441c48ee49be4" exitCode=0 Oct 30 05:52:22.467426 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.467380 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" event={"ID":"8eacbef0-b3e6-419d-83a4-64ddd49f1d93","Type":"ContainerDied","Data":"c8cc167164c721b991effc2f03897ae2be0f0c4cf72abe2e509441c48ee49be4"} Oct 30 05:52:22.471144 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.471123 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:22.471317 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.471300 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" event={"ID":"8eacbef0-b3e6-419d-83a4-64ddd49f1d93","Type":"ContainerDied","Data":"b2971d383e70e5731f2e47518604c6178f55712594acca130bbb1c27d76914ac"} Oct 30 05:52:22.471425 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.471411 5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2971d383e70e5731f2e47518604c6178f55712594acca130bbb1c27d76914ac" Oct 30 05:52:22.479232 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.475324 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:52:22.776106 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.776064 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.892485 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.892589 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-data\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.892646 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istiod-ca-cert\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.892688 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-token\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-token\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.892851 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-envoy\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.892954 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-credential-socket\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.893028 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.893131 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-certs\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.893176 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-kube-api-access-l7nqk\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-kube-api-access-l7nqk\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:22.919539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.909999 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-credential-socket" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:22.933571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.929355 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-data" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:22.933571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.929772 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istiod-ca-cert" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:22.933571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.930014 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-certs" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:22.954715 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.948121 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m"] Oct 30 05:52:22.954715 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.949197 5072 cpu_manager.go:414] "containerMap: removing container" podUID="8eacbef0-b3e6-419d-83a4-64ddd49f1d93" containerName="istio-proxy" Oct 30 05:52:22.954715 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.949217 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eacbef0-b3e6-419d-83a4-64ddd49f1d93" containerName="istio-proxy" Oct 30 05:52:22.954715 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:22.949395 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eacbef0-b3e6-419d-83a4-64ddd49f1d93" containerName="istio-proxy" Oct 30 05:52:23.010503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.001231 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.010503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.003565 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:23.010503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.006442 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:52:23.010503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.006509 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m"] Oct 30 05:52:23.043593 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.020870 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-d96a1f-openshift-default-dockercfg-whxzd\"" Oct 30 05:52:23.045361 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.045331 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:23.045845 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:52:23.045797 5072 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8eacbef0-b3e6-419d-83a4-64ddd49f1d93/volumes/kubernetes.io~empty-dir/workload-socket Oct 30 05:52:23.045845 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.045817 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:23.046123 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.046088 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo\") pod \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\" (UID: \"8eacbef0-b3e6-419d-83a4-64ddd49f1d93\") " Oct 30 05:52:23.046722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.046698 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-envoy\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.046944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.046926 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-token\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.059319 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:52:23.050251 5072 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8eacbef0-b3e6-419d-83a4-64ddd49f1d93/volumes/kubernetes.io~downward-api/istio-podinfo Oct 30 05:52:23.059643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.059623 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:52:23.059868 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.059583 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-certs\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.060036 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.060016 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4a3b88-8b55-4812-b640-df363c7e89d4-istiod-ca-cert\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.061182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.061139 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrl67\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-kube-api-access-nrl67\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.061830 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.061804 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-podinfo\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.078329 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.078290 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-socket\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.087682 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.087647 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-data\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.088338 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.088316 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-credential-socket\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.132118 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.131992 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:23.132310 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.132294 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:23.132448 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.132433 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:23.132658 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.132646 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:23.132772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.132761 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:23.132929 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.132917 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:23.182483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.177927 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-kube-api-access-l7nqk" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "kube-api-access-l7nqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:23.183790 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.183731 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-token" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:23.189879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.189848 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-envoy" pod "8eacbef0-b3e6-419d-83a4-64ddd49f1d93" (UID: "8eacbef0-b3e6-419d-83a4-64ddd49f1d93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:23.235650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.235598 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrl67\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-kube-api-access-nrl67\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.235985 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.235956 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-podinfo\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.236260 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.236237 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-socket\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.236435 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.236416 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-data\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.236635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.236617 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-credential-socket\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.236826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.236807 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-envoy\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.236997 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.236978 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-token\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.237139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.237122 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-certs\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.237266 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.237251 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4a3b88-8b55-4812-b640-df363c7e89d4-istiod-ca-cert\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.237494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.237476 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:23.237622 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.237607 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nqk\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-kube-api-access-l7nqk\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:23.237722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.237709 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eacbef0-b3e6-419d-83a4-64ddd49f1d93-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:23.238607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.238588 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4a3b88-8b55-4812-b640-df363c7e89d4-istiod-ca-cert\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.239028 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.239008 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-socket\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.239421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.239399 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-data\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.242072 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.240622 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-credential-socket\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.242210 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.241112 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-certs\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.247777 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.245867 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-podinfo\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.248062 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.248024 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-envoy\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.260194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.260139 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrl67\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-kube-api-access-nrl67\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.266928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.266902 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-token\") pod \"e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.475197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.475163 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq" Oct 30 05:52:23.493059 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.493029 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:23.626566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.609071 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq"] Oct 30 05:52:23.668494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:23.667572 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3224b039-openshift-default-845595b5d9-gfdmq"] Oct 30 05:52:24.188477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:24.188426 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m"] Oct 30 05:52:24.191207 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:52:24.191181 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4a3b88_8b55_4812_b640_df363c7e89d4.slice/crio-d81a2d976acadb8149eef64a8f8d6b56b3267c4c77c0bfb59fd443d0b385986a WatchSource:0}: Error finding container d81a2d976acadb8149eef64a8f8d6b56b3267c4c77c0bfb59fd443d0b385986a: Status 404 returned error can't find the container with id d81a2d976acadb8149eef64a8f8d6b56b3267c4c77c0bfb59fd443d0b385986a Oct 30 05:52:24.194775 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:24.194739 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:24.194853 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:24.194821 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:24.194897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:24.194864 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:24.485772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:24.485665 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" event={"ID":"9b4a3b88-8b55-4812-b640-df363c7e89d4","Type":"ContainerStarted","Data":"5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd"} Oct 30 05:52:24.485772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:24.485744 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" event={"ID":"9b4a3b88-8b55-4812-b640-df363c7e89d4","Type":"ContainerStarted","Data":"d81a2d976acadb8149eef64a8f8d6b56b3267c4c77c0bfb59fd443d0b385986a"} Oct 30 05:52:24.494234 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:24.494207 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:24.521299 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:24.521274 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eacbef0-b3e6-419d-83a4-64ddd49f1d93" path="/var/lib/kubelet/pods/8eacbef0-b3e6-419d-83a4-64ddd49f1d93/volumes" Oct 30 05:52:24.595800 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:24.595754 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podStartSLOduration=2.595737362 podStartE2EDuration="2.595737362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:24.592156036 +0000 UTC m=+1304.195845294" watchObservedRunningTime="2026-10-30 05:52:24.595737362 +0000 UTC m=+1304.199426618" Oct 30 05:52:25.261143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.261116 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:52:25.365271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.364359 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-credential-socket\") pod \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " Oct 30 05:52:25.365271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.364518 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-certs\") pod \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " Oct 30 05:52:25.365271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.364586 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-kube-api-access-fl2w9\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-kube-api-access-fl2w9\") pod \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " Oct 30 05:52:25.365271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.364621 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-token\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-token\") pod \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " Oct 30 05:52:25.365271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.364645 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-socket\") pod \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " Oct 30 05:52:25.365271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.364738 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-envoy\") pod \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " Oct 30 05:52:25.365271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.364777 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-data\") pod \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " Oct 30 05:52:25.365271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.364796 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istiod-ca-cert\") pod \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " Oct 30 05:52:25.365271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.364814 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-podinfo\") pod \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\" (UID: \"1aab9b7f-029d-4619-9cbc-0f59084d6d20\") " Oct 30 05:52:25.366809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.366755 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-credential-socket" pod "1aab9b7f-029d-4619-9cbc-0f59084d6d20" (UID: "1aab9b7f-029d-4619-9cbc-0f59084d6d20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.367073 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.367051 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-certs" pod "1aab9b7f-029d-4619-9cbc-0f59084d6d20" (UID: "1aab9b7f-029d-4619-9cbc-0f59084d6d20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.369380 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.369346 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-data" pod "1aab9b7f-029d-4619-9cbc-0f59084d6d20" (UID: "1aab9b7f-029d-4619-9cbc-0f59084d6d20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.369887 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.369862 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-podinfo" pod "1aab9b7f-029d-4619-9cbc-0f59084d6d20" (UID: "1aab9b7f-029d-4619-9cbc-0f59084d6d20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:52:25.370265 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.370240 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istiod-ca-cert" pod "1aab9b7f-029d-4619-9cbc-0f59084d6d20" (UID: "1aab9b7f-029d-4619-9cbc-0f59084d6d20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:25.370447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.370426 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-socket" pod "1aab9b7f-029d-4619-9cbc-0f59084d6d20" (UID: "1aab9b7f-029d-4619-9cbc-0f59084d6d20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.379805 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.379777 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-kube-api-access-fl2w9" pod "1aab9b7f-029d-4619-9cbc-0f59084d6d20" (UID: "1aab9b7f-029d-4619-9cbc-0f59084d6d20"). InnerVolumeSpecName "kube-api-access-fl2w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:25.389728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.389694 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-token" pod "1aab9b7f-029d-4619-9cbc-0f59084d6d20" (UID: "1aab9b7f-029d-4619-9cbc-0f59084d6d20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:25.390268 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.390236 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-envoy" pod "1aab9b7f-029d-4619-9cbc-0f59084d6d20" (UID: "1aab9b7f-029d-4619-9cbc-0f59084d6d20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.470179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.468449 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.470179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.469168 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl2w9\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-kube-api-access-fl2w9\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.470179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.469342 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.470179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.469362 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.470179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.469376 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.470179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.469389 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.470179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.469405 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.470179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.469418 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab9b7f-029d-4619-9cbc-0f59084d6d20-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.470179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.469433 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab9b7f-029d-4619-9cbc-0f59084d6d20-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.495798 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.495753 5072 generic.go:356] "Generic (PLEG): container finished" podID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerID="e82d050862f95b8bd0df1612d8b818e04bb147f679b32b2269193ad700d98bca" exitCode=0 Oct 30 05:52:25.496237 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.496072 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" event={"ID":"052f4db2-8671-4a39-aa23-cdd244c4288e","Type":"ContainerDied","Data":"e82d050862f95b8bd0df1612d8b818e04bb147f679b32b2269193ad700d98bca"} Oct 30 05:52:25.496237 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.496167 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" event={"ID":"052f4db2-8671-4a39-aa23-cdd244c4288e","Type":"ContainerDied","Data":"ad563740a9f143dd14ff574919312010b01ae309dcc9641fd78400572f700652"} Oct 30 05:52:25.496237 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.496191 5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad563740a9f143dd14ff574919312010b01ae309dcc9641fd78400572f700652" Oct 30 05:52:25.498179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.498154 5072 generic.go:356] "Generic (PLEG): container finished" podID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerID="1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a" exitCode=0 Oct 30 05:52:25.498316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.498233 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" Oct 30 05:52:25.498316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.498239 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" event={"ID":"1aab9b7f-029d-4619-9cbc-0f59084d6d20","Type":"ContainerDied","Data":"1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a"} Oct 30 05:52:25.498316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.498287 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm" event={"ID":"1aab9b7f-029d-4619-9cbc-0f59084d6d20","Type":"ContainerDied","Data":"537848c91ca92fe3c65ba3a327227ebd905ee5749b45f48f51e65f60e0581936"} Oct 30 05:52:25.498316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.498310 5072 scope.go:122] "RemoveContainer" containerID="1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a" Oct 30 05:52:25.500390 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.500370 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:25.508051 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.508030 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:52:25.515819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.515802 5072 scope.go:122] "RemoveContainer" containerID="1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a" Oct 30 05:52:25.516105 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:52:25.516076 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a\": container with ID starting with 1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a not found: ID does not exist" containerID="1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a" Oct 30 05:52:25.516163 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.516115 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a"} err="failed to get container status \"1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a\": rpc error: code = NotFound desc = could not find container \"1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a\": container with ID starting with 1d24c97b1d28f713944a4bc9f0e36ffb4882ae75588086884d5fd2f8bd88c51a not found: ID does not exist" Oct 30 05:52:25.643251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.643222 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm"] Oct 30 05:52:25.651366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.651339 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-5qfjm"] Oct 30 05:52:25.672705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.672678 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-envoy\") pod \"052f4db2-8671-4a39-aa23-cdd244c4288e\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " Oct 30 05:52:25.672861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.672765 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-credential-socket\") pod \"052f4db2-8671-4a39-aa23-cdd244c4288e\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " Oct 30 05:52:25.672861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.672836 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-token\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-token\") pod \"052f4db2-8671-4a39-aa23-cdd244c4288e\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " Oct 30 05:52:25.673203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.673184 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-socket\") pod \"052f4db2-8671-4a39-aa23-cdd244c4288e\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " Oct 30 05:52:25.673358 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.673344 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-podinfo\") pod \"052f4db2-8671-4a39-aa23-cdd244c4288e\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " Oct 30 05:52:25.673809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.673791 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-data\") pod \"052f4db2-8671-4a39-aa23-cdd244c4288e\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " Oct 30 05:52:25.674000 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.673194 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-credential-socket" pod "052f4db2-8671-4a39-aa23-cdd244c4288e" (UID: "052f4db2-8671-4a39-aa23-cdd244c4288e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.674192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.673430 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-socket" pod "052f4db2-8671-4a39-aa23-cdd244c4288e" (UID: "052f4db2-8671-4a39-aa23-cdd244c4288e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.674192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.674123 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-data" pod "052f4db2-8671-4a39-aa23-cdd244c4288e" (UID: "052f4db2-8671-4a39-aa23-cdd244c4288e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.674501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.674346 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052f4db2-8671-4a39-aa23-cdd244c4288e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052f4db2-8671-4a39-aa23-cdd244c4288e-istiod-ca-cert\") pod \"052f4db2-8671-4a39-aa23-cdd244c4288e\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " Oct 30 05:52:25.674604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.674499 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-kube-api-access-zhmsl\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-kube-api-access-zhmsl\") pod \"052f4db2-8671-4a39-aa23-cdd244c4288e\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " Oct 30 05:52:25.674604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.674550 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-certs\") pod \"052f4db2-8671-4a39-aa23-cdd244c4288e\" (UID: \"052f4db2-8671-4a39-aa23-cdd244c4288e\") " Oct 30 05:52:25.675317 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.675291 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.675411 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.675320 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.675411 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.675335 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.675590 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.675519 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-envoy" pod "052f4db2-8671-4a39-aa23-cdd244c4288e" (UID: "052f4db2-8671-4a39-aa23-cdd244c4288e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.675758 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.675724 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-certs" pod "052f4db2-8671-4a39-aa23-cdd244c4288e" (UID: "052f4db2-8671-4a39-aa23-cdd244c4288e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:25.676771 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.676750 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052f4db2-8671-4a39-aa23-cdd244c4288e-istiod-ca-cert" pod "052f4db2-8671-4a39-aa23-cdd244c4288e" (UID: "052f4db2-8671-4a39-aa23-cdd244c4288e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:25.683785 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.683648 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-token" pod "052f4db2-8671-4a39-aa23-cdd244c4288e" (UID: "052f4db2-8671-4a39-aa23-cdd244c4288e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:25.683785 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.683755 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-podinfo" pod "052f4db2-8671-4a39-aa23-cdd244c4288e" (UID: "052f4db2-8671-4a39-aa23-cdd244c4288e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:52:25.684137 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.684096 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-kube-api-access-zhmsl" pod "052f4db2-8671-4a39-aa23-cdd244c4288e" (UID: "052f4db2-8671-4a39-aa23-cdd244c4288e"). InnerVolumeSpecName "kube-api-access-zhmsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:25.787091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.786627 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.787091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.786667 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.787091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.786682 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052f4db2-8671-4a39-aa23-cdd244c4288e-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.787091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.786697 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052f4db2-8671-4a39-aa23-cdd244c4288e-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.787091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.786713 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhmsl\" (UniqueName: \"kubernetes.io/projected/052f4db2-8671-4a39-aa23-cdd244c4288e-kube-api-access-zhmsl\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:25.787091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:25.786729 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052f4db2-8671-4a39-aa23-cdd244c4288e-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:52:26.539509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:26.536573 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv" Oct 30 05:52:26.541172 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:26.541146 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" path="/var/lib/kubelet/pods/1aab9b7f-029d-4619-9cbc-0f59084d6d20/volumes" Oct 30 05:52:26.541936 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:26.541917 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:26.542066 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:26.542001 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:52:26.601580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:26.601550 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv"] Oct 30 05:52:26.612717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:26.612688 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-stjqv"] Oct 30 05:52:28.524148 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:28.524114 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" path="/var/lib/kubelet/pods/052f4db2-8671-4a39-aa23-cdd244c4288e/volumes" Oct 30 05:52:29.149553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.149505 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25"] Oct 30 05:52:29.150734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.150707 5072 cpu_manager.go:414] "containerMap: removing container" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" Oct 30 05:52:29.150734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.150732 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" Oct 30 05:52:29.150941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.150751 5072 cpu_manager.go:414] "containerMap: removing container" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" Oct 30 05:52:29.150941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.150759 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" Oct 30 05:52:29.150941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.150914 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aab9b7f-029d-4619-9cbc-0f59084d6d20" containerName="istio-proxy" Oct 30 05:52:29.150941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.150931 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="052f4db2-8671-4a39-aa23-cdd244c4288e" containerName="istio-proxy" Oct 30 05:52:29.156560 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.156536 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:29.268532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.260977 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d96a1f-dockercfg-9qkvg\"" Oct 30 05:52:29.285844 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.285806 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d96a1f\"" Oct 30 05:52:29.510773 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.510682 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08171d10-1ddb-4879-bfb6-851928e94d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25\" (UID: \"08171d10-1ddb-4879-bfb6-851928e94d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:29.510958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.510938 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pt2\" (UniqueName: \"kubernetes.io/projected/08171d10-1ddb-4879-bfb6-851928e94d79-kube-api-access-h6pt2\") pod \"payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25\" (UID: \"08171d10-1ddb-4879-bfb6-851928e94d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:29.524570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.524529 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25"] Oct 30 05:52:29.621547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.621495 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08171d10-1ddb-4879-bfb6-851928e94d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25\" (UID: \"08171d10-1ddb-4879-bfb6-851928e94d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:29.630978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.630939 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pt2\" (UniqueName: \"kubernetes.io/projected/08171d10-1ddb-4879-bfb6-851928e94d79-kube-api-access-h6pt2\") pod \"payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25\" (UID: \"08171d10-1ddb-4879-bfb6-851928e94d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:29.632475 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.628810 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08171d10-1ddb-4879-bfb6-851928e94d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25\" (UID: \"08171d10-1ddb-4879-bfb6-851928e94d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:29.658176 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.658146 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pt2\" (UniqueName: \"kubernetes.io/projected/08171d10-1ddb-4879-bfb6-851928e94d79-kube-api-access-h6pt2\") pod \"payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25\" (UID: \"08171d10-1ddb-4879-bfb6-851928e94d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:29.775491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.775395 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:29.868876 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.868830 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj"] Oct 30 05:52:29.876522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.876424 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:29.908953 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.908918 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"distinct-2-w0-a14091-kserve-self-signed-certs\"" Oct 30 05:52:29.922728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.922699 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj"] Oct 30 05:52:29.960860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.959252 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-home\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:29.960860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.959303 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-model-cache\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:29.960860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.959352 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-dshm\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:29.960860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.959511 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tls-certs\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:29.960860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.959541 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tmp-dir\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:29.960860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:29.959621 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q6tx\" (UniqueName: \"kubernetes.io/projected/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-kube-api-access-7q6tx\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.082529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.078797 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q6tx\" (UniqueName: \"kubernetes.io/projected/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-kube-api-access-7q6tx\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.082529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.079111 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-home\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.082529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.079148 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-model-cache\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.082529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.079188 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-dshm\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.082529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.079309 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tls-certs\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.082529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.079333 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tmp-dir\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.082529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.079856 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tmp-dir\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.082529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.080697 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-home\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.082529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.081009 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-model-cache\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.140134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.135116 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q6tx\" (UniqueName: \"kubernetes.io/projected/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-kube-api-access-7q6tx\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.140134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.136064 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tls-certs\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.140134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.137872 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-dshm\") pod \"distinct-2-w0-a14091-kserve-8f49f7666-vx9fj\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.186885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.186847 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25"] Oct 30 05:52:30.202032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.201947 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:30.686086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.681024 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:30.686086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.681102 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" event={"ID":"08171d10-1ddb-4879-bfb6-851928e94d79","Type":"ContainerStarted","Data":"6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce"} Oct 30 05:52:30.686086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.681132 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" event={"ID":"08171d10-1ddb-4879-bfb6-851928e94d79","Type":"ContainerStarted","Data":"b75f921670d09b6a248421ffb613b30610cb699b5af0dd8a842913cec436728e"} Oct 30 05:52:30.729257 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.729126 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" podStartSLOduration=1.729104177 podStartE2EDuration="1.729104177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:30.719063213 +0000 UTC m=+1310.322752470" watchObservedRunningTime="2026-10-30 05:52:30.729104177 +0000 UTC m=+1310.332793434" Oct 30 05:52:30.729932 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:52:30.729901 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e70915b_1cdf_4270_81eb_5c7c8f9a5313.slice/crio-11462be43640f43d21ad30b85959356cc275321c5752e67e6f2d765ab5057a3d WatchSource:0}: Error finding container 11462be43640f43d21ad30b85959356cc275321c5752e67e6f2d765ab5057a3d: Status 404 returned error can't find the container with id 11462be43640f43d21ad30b85959356cc275321c5752e67e6f2d765ab5057a3d Oct 30 05:52:30.738247 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:30.738219 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj"] Oct 30 05:52:30.753004 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:52:30.752968 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:52:31.659530 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:31.659493 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" event={"ID":"9e70915b-1cdf-4270-81eb-5c7c8f9a5313","Type":"ContainerStarted","Data":"894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb"} Oct 30 05:52:31.659701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:31.659546 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" event={"ID":"9e70915b-1cdf-4270-81eb-5c7c8f9a5313","Type":"ContainerStarted","Data":"11462be43640f43d21ad30b85959356cc275321c5752e67e6f2d765ab5057a3d"} Oct 30 05:52:31.749659 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:31.749598 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podStartSLOduration=2.749576658 podStartE2EDuration="2.749576658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:31.745779852 +0000 UTC m=+1311.349469162" watchObservedRunningTime="2026-10-30 05:52:31.749576658 +0000 UTC m=+1311.353265916" Oct 30 05:52:32.202780 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:32.202748 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:33.569153 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.569110 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82"] Oct 30 05:52:33.573842 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.573817 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.601269 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.601246 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a14091-openshift-default-dockercfg-xdwgh\"" Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.628554 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82"] Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.633617 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba108819-39d4-4da9-866b-50ba6ccbfc54-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.633714 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.633776 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.633815 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwlm\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-kube-api-access-rvwlm\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.633869 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.633921 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.633944 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.633975 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.638134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.634114 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.736374 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.736329 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.736609 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.736561 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba108819-39d4-4da9-866b-50ba6ccbfc54-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.736684 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.736660 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737007 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.736938 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737007 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.736998 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwlm\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-kube-api-access-rvwlm\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.737023 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.737078 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.737140 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.737164 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.737192 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737472 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.737368 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737472 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.737388 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737568 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.737544 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba108819-39d4-4da9-866b-50ba6ccbfc54-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.737615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.737572 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.739746 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.739719 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.739992 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.739975 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.760263 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.760237 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwlm\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-kube-api-access-rvwlm\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.762339 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.762316 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:33.888766 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:33.888739 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:34.221129 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:34.221102 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82"] Oct 30 05:52:34.222012 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:52:34.221980 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba108819_39d4_4da9_866b_50ba6ccbfc54.slice/crio-e85652b63bac10bbe009114b35027726d6d8358defe218b2b9570e0650ee1a98 WatchSource:0}: Error finding container e85652b63bac10bbe009114b35027726d6d8358defe218b2b9570e0650ee1a98: Status 404 returned error can't find the container with id e85652b63bac10bbe009114b35027726d6d8358defe218b2b9570e0650ee1a98 Oct 30 05:52:34.225720 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:34.225680 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:34.225830 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:34.225767 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:34.225830 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:34.225809 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:52:34.695762 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:34.695725 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" event={"ID":"ba108819-39d4-4da9-866b-50ba6ccbfc54","Type":"ContainerStarted","Data":"6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320"} Oct 30 05:52:34.696256 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:34.695781 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" event={"ID":"ba108819-39d4-4da9-866b-50ba6ccbfc54","Type":"ContainerStarted","Data":"e85652b63bac10bbe009114b35027726d6d8358defe218b2b9570e0650ee1a98"} Oct 30 05:52:34.745498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:34.745427 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podStartSLOduration=1.745412972 podStartE2EDuration="1.745412972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:34.74139413 +0000 UTC m=+1314.345083390" watchObservedRunningTime="2026-10-30 05:52:34.745412972 +0000 UTC m=+1314.349102290" Oct 30 05:52:34.893484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:34.889568 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:35.891132 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:35.891100 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:52:35.891581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:35.891174 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 05:52:36.455385 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.455354 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54"] Oct 30 05:52:36.463114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.463090 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.500241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.500169 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-0b481b-serving-cert\"" Oct 30 05:52:36.506866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.506836 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54"] Oct 30 05:52:36.507387 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.507366 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2419935d-aa24-4e93-a1b4-2a20923b77df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.507440 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.507403 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.507734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.507712 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.507838 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.507816 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrss\" (UniqueName: \"kubernetes.io/projected/2419935d-aa24-4e93-a1b4-2a20923b77df-kube-api-access-chrss\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.627201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.609780 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2419935d-aa24-4e93-a1b4-2a20923b77df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.627201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.609841 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.627201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.610070 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.627201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.610176 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrss\" (UniqueName: \"kubernetes.io/projected/2419935d-aa24-4e93-a1b4-2a20923b77df-kube-api-access-chrss\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.627201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.611477 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2419935d-aa24-4e93-a1b4-2a20923b77df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.627201 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:52:36.611582 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-serving-cert: secret "maas-api-e2e-shared-a-w4-0b481b-serving-cert" not found Oct 30 05:52:36.627201 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:52:36.611638 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls podName:2419935d-aa24-4e93-a1b4-2a20923b77df nodeName:}" failed. No retries permitted until 2026-10-30 05:52:37.111619332 +0000 UTC m=+1316.715308570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls") pod "maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" (UID: "2419935d-aa24-4e93-a1b4-2a20923b77df") : secret "maas-api-e2e-shared-a-w4-0b481b-serving-cert" not found Oct 30 05:52:36.652667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.633051 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.665489 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.665419 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrss\" (UniqueName: \"kubernetes.io/projected/2419935d-aa24-4e93-a1b4-2a20923b77df-kube-api-access-chrss\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:36.918909 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.914604 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:52:36.918909 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:36.914660 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 05:52:37.209557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:37.209324 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:37.218170 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:37.218127 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:37.382483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:37.379518 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:37.879483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:37.875968 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54"] Oct 30 05:52:37.881631 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:52:37.881595 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2419935d_aa24_4e93_a1b4_2a20923b77df.slice/crio-6b754980e43c1226e888eee1d558f32a6be2b3d1f68706c55c59e4f2f5817c44 WatchSource:0}: Error finding container 6b754980e43c1226e888eee1d558f32a6be2b3d1f68706c55c59e4f2f5817c44: Status 404 returned error can't find the container with id 6b754980e43c1226e888eee1d558f32a6be2b3d1f68706c55c59e4f2f5817c44 Oct 30 05:52:37.890903 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:37.890861 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:52:37.891123 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:37.891092 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 05:52:38.808785 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:38.808747 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" event={"ID":"2419935d-aa24-4e93-a1b4-2a20923b77df","Type":"ContainerStarted","Data":"d601d8a43653875d19a29926ead4e7c8621b7da52c12d4a2fd54c10fff5573dd"} Oct 30 05:52:38.820673 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:38.820638 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" event={"ID":"2419935d-aa24-4e93-a1b4-2a20923b77df","Type":"ContainerStarted","Data":"6b754980e43c1226e888eee1d558f32a6be2b3d1f68706c55c59e4f2f5817c44"} Oct 30 05:52:38.822016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:38.821994 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:38.870831 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:38.870769 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" podStartSLOduration=2.420578099 podStartE2EDuration="2.870749092s" totalImagesPullingTime="450.170993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:38.868828709 +0000 UTC m=+1318.472517989" watchObservedRunningTime="2026-10-30 05:52:38.870749092 +0000 UTC m=+1318.474438350" Oct 30 05:52:38.889720 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:38.889682 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:52:38.889884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:38.889747 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 05:52:39.890091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:39.890060 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:52:39.890534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:39.890117 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 05:52:40.202989 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.202907 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:40.279297 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.279264 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:40.384764 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.384659 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss"] Oct 30 05:52:40.389815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.389789 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.454181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.454104 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8bab43d7-serving-cert\"" Oct 30 05:52:40.457084 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.457063 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss"] Oct 30 05:52:40.560803 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.560772 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-metrics-certs\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.561026 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.561011 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b909d1-426d-4d37-8324-4b01aa536802-openshift-service-ca\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.561151 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.561035 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg8c\" (UniqueName: \"kubernetes.io/projected/46b909d1-426d-4d37-8324-4b01aa536802-kube-api-access-bsg8c\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.561151 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.561081 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-maas-api-tls\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.668486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.667685 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b909d1-426d-4d37-8324-4b01aa536802-openshift-service-ca\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.668486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.667736 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg8c\" (UniqueName: \"kubernetes.io/projected/46b909d1-426d-4d37-8324-4b01aa536802-kube-api-access-bsg8c\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.668486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.667830 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-maas-api-tls\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.668486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.668006 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-metrics-certs\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.677481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.673648 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b909d1-426d-4d37-8324-4b01aa536802-openshift-service-ca\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.696561 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.696041 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8bab43d7-serving-cert\"" Oct 30 05:52:40.757581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.722733 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg8c\" (UniqueName: \"kubernetes.io/projected/46b909d1-426d-4d37-8324-4b01aa536802-kube-api-access-bsg8c\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.771120 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.771084 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-metrics-certs\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.801433 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.779101 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-maas-api-tls\") pod \"maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:40.889615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.889178 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:52:40.889615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:40.889235 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 05:52:41.008636 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:41.008552 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:41.040317 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:41.040287 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:52:41.292647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:41.292554 5072 scope.go:122] "RemoveContainer" containerID="e82d050862f95b8bd0df1612d8b818e04bb147f679b32b2269193ad700d98bca" Oct 30 05:52:41.405583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:41.405537 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss"] Oct 30 05:52:41.417001 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:52:41.416960 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b909d1_426d_4d37_8324_4b01aa536802.slice/crio-4d8d29e0bfe5f63de7da10107a6dc915cdb3e31671d4a1c7a27b103d1fed3ea8 WatchSource:0}: Error finding container 4d8d29e0bfe5f63de7da10107a6dc915cdb3e31671d4a1c7a27b103d1fed3ea8: Status 404 returned error can't find the container with id 4d8d29e0bfe5f63de7da10107a6dc915cdb3e31671d4a1c7a27b103d1fed3ea8 Oct 30 05:52:41.703485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:41.699787 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 05:52:42.095567 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:42.012134 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:52:42.095567 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:42.012234 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 05:52:42.127159 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:42.127132 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" event={"ID":"46b909d1-426d-4d37-8324-4b01aa536802","Type":"ContainerStarted","Data":"4d8d29e0bfe5f63de7da10107a6dc915cdb3e31671d4a1c7a27b103d1fed3ea8"} Oct 30 05:52:43.218482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:43.205704 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" event={"ID":"46b909d1-426d-4d37-8324-4b01aa536802","Type":"ContainerStarted","Data":"808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2"} Oct 30 05:52:43.218482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:43.206985 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:43.291268 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:43.291208 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" podStartSLOduration=2.634022301 podStartE2EDuration="3.291187973s" totalImagesPullingTime="657.165672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:43.274186949 +0000 UTC m=+1322.877876233" watchObservedRunningTime="2026-10-30 05:52:43.291187973 +0000 UTC m=+1322.894877230" Oct 30 05:52:43.301355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:43.300724 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:43.302348 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:43.302316 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:43.305134 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:43.305114 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 05:52:45.903876 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:45.903846 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 05:52:46.315426 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.315347 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf"] Oct 30 05:52:46.320273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.320251 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.339086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.339044 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-d96a1f-serving-cert\"" Oct 30 05:52:46.342249 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.342209 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf"] Oct 30 05:52:46.385402 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.385365 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhdm\" (UniqueName: \"kubernetes.io/projected/38da4d60-a766-494c-a6f0-91cc3577464c-kube-api-access-dxhdm\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.385594 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.385421 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.385683 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.385659 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.385891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.385871 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38da4d60-a766-494c-a6f0-91cc3577464c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.487328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.487296 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhdm\" (UniqueName: \"kubernetes.io/projected/38da4d60-a766-494c-a6f0-91cc3577464c-kube-api-access-dxhdm\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.487532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.487341 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.487532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.487441 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.487679 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.487658 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38da4d60-a766-494c-a6f0-91cc3577464c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.488334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.488290 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38da4d60-a766-494c-a6f0-91cc3577464c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.489951 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.489928 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.490083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.489979 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.511067 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.511039 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhdm\" (UniqueName: \"kubernetes.io/projected/38da4d60-a766-494c-a6f0-91cc3577464c-kube-api-access-dxhdm\") pod \"maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:46.644571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:46.643106 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:47.062499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:47.062469 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf"] Oct 30 05:52:47.062823 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:52:47.062623 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38da4d60_a766_494c_a6f0_91cc3577464c.slice/crio-82225f493f5a93c99d47a0f7a742100d511f1c1bbc875f885fbaf2918549c72d WatchSource:0}: Error finding container 82225f493f5a93c99d47a0f7a742100d511f1c1bbc875f885fbaf2918549c72d: Status 404 returned error can't find the container with id 82225f493f5a93c99d47a0f7a742100d511f1c1bbc875f885fbaf2918549c72d Oct 30 05:52:47.256751 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:47.255280 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" event={"ID":"38da4d60-a766-494c-a6f0-91cc3577464c","Type":"ContainerStarted","Data":"82225f493f5a93c99d47a0f7a742100d511f1c1bbc875f885fbaf2918549c72d"} Oct 30 05:52:48.265251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:48.265207 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" event={"ID":"38da4d60-a766-494c-a6f0-91cc3577464c","Type":"ContainerStarted","Data":"e35222d0a1c183ce5734ae22ed399a74d59e73dec0c13a49725ac5949f87fb2e"} Oct 30 05:52:48.265850 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:48.265448 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:52:48.326785 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:48.326697 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" podStartSLOduration=1.884985807 podStartE2EDuration="2.326675264s" totalImagesPullingTime="441.689457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:48.321605768 +0000 UTC m=+1327.925295025" watchObservedRunningTime="2026-10-30 05:52:48.326675264 +0000 UTC m=+1327.930364523" Oct 30 05:52:50.262798 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:50.262769 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:52:54.279172 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:52:54.279140 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 05:53:00.713938 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:53:00.713890 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:53:04.958672 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:04.958635 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g"] Oct 30 05:53:04.970022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:04.969997 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.013249 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.013211 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g"] Oct 30 05:53:05.127981 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.127938 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.128165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.128076 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84c2m\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-kube-api-access-84c2m\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.128255 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.128240 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.128305 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.128292 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.128427 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.128403 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.128479 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.128445 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.128553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.128533 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.128694 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.128619 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.128694 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.128661 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230028 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230089 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230151 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230175 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230211 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230343 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230430 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84c2m\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-kube-api-access-84c2m\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230621 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230732 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230677 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.230856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.230835 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.231156 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.231131 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.231292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.231212 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.231292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.231252 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.231292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.231262 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.233917 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.233877 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.234038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.233959 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.253689 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.253638 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.254161 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.254137 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84c2m\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-kube-api-access-84c2m\") pod \"maas-default-gateway-openshift-default-84587756dc-zdt2g\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.329375 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.329341 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:05.808503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.788733 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g"] Oct 30 05:53:05.855031 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:53:05.854838 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77a1aab_55dd_47d6_a80c_c5ea3df300de.slice/crio-6d33aeaba4bfb9c4d0049c0c5d9794053f055b730b0c7f423763de60ac815bac WatchSource:0}: Error finding container 6d33aeaba4bfb9c4d0049c0c5d9794053f055b730b0c7f423763de60ac815bac: Status 404 returned error can't find the container with id 6d33aeaba4bfb9c4d0049c0c5d9794053f055b730b0c7f423763de60ac815bac Oct 30 05:53:05.858652 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.858615 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:05.858887 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.858858 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:05.859043 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:05.859021 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:06.865650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:06.865615 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" event={"ID":"e77a1aab-55dd-47d6-a80c-c5ea3df300de","Type":"ContainerStarted","Data":"a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1"} Oct 30 05:53:06.866032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:06.865678 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" event={"ID":"e77a1aab-55dd-47d6-a80c-c5ea3df300de","Type":"ContainerStarted","Data":"6d33aeaba4bfb9c4d0049c0c5d9794053f055b730b0c7f423763de60ac815bac"} Oct 30 05:53:06.933483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:06.932902 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podStartSLOduration=2.932881849 podStartE2EDuration="2.932881849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:06.929930712 +0000 UTC m=+1346.533619969" watchObservedRunningTime="2026-10-30 05:53:06.932881849 +0000 UTC m=+1346.536571107" Oct 30 05:53:07.335538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:07.335360 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:07.335722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:07.335556 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:07.335722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:07.335602 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:08.342022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:08.341987 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:08.342681 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:08.342648 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:09.330886 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:09.330847 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:09.332486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:09.332432 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:10.332070 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:10.332030 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:10.343622 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:10.343572 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:11.345638 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:11.345602 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:11.346753 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:11.346717 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:12.339660 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:12.339613 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:12.339979 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:12.339946 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:13.354488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:13.340088 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:13.354488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:13.340163 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:13.806593 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:13.782441 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:13.903613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:13.903564 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:14.495069 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.495015 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:14.497768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.497422 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:14.497768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.497514 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:14.516386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.516331 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:53:14.595025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.525641 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:53:14.595025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.585003 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:14.702470 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.702409 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:14.780939 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.770314 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:53:14.807486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.785339 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:53:14.807931 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.807889 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:53:14.909620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.909570 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:14.926608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:14.926571 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:53:15.365832 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:15.364891 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:53:15.491276 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:15.491234 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:15.491609 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:15.491574 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:15.506590 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:15.506541 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 05:53:15.515801 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:15.515767 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:53:15.609509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:15.608014 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:15.609509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:15.608480 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:53:15.775636 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:15.775553 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:53:15.798429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:15.798363 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:16.375602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:16.369389 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:16.381328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:16.381295 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:16.382180 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:16.382149 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:16.623977 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:16.622290 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:16.772540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:16.767212 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:16.821730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:16.821684 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:17.338486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:17.337583 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:17.338486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:17.337649 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:17.371839 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:17.371783 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:17.372586 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:17.372547 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:53:17.708495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:17.706381 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:17.795490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:17.794787 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:17.795490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:17.795492 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:53:17.828485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:17.826043 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:18.054540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:18.043115 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Oct 30 05:53:18.331177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:18.331084 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:18.331495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:18.331412 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:18.375926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:18.375145 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:53:18.712480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:18.712258 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:53:18.761484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:18.751013 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:53:18.792493 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:18.792419 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:18.806909 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:18.795552 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:19.123539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:19.106891 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:19.415546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:19.410957 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 05:53:19.415546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:19.411022 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 05:53:19.489485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:19.447032 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:19.796690 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:19.796560 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:53:19.894719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:19.832686 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:19.895337 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:19.832722 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:19.895890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:19.895868 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:53:20.402958 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:20.402834 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:20.738603 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:20.734928 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:20.931508 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:20.931151 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:53:20.938481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:20.932237 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 05:53:21.151999 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.151960 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:53:21.208494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.186858 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:53:21.439880 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.439773 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Oct 30 05:53:21.482622 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.482581 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:53:21.482929 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.482896 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:21.750547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.741670 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:53:21.957290 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.957245 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:21.982484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.957415 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:53:21.983012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.957440 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:53:22.017591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:21.983445 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:53:22.162784 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.162677 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:53:22.610845 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.610809 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:53:22.611115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.611088 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:22.795491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.791754 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:22.804796 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.801520 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:53:22.841491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.833744 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:53:22.868581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.850770 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:22.873895 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.873865 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 05:53:22.877534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.876767 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:53:22.879133 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:22.878398 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:53:27.587199 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:27.587159 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:53:27.588527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:27.588494 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:29.951570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.951539 5072 generic.go:356] "Generic (PLEG): container finished" podID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerID="5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd" exitCode=0 Oct 30 05:53:29.952270 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.951688 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" event={"ID":"9b4a3b88-8b55-4812-b640-df363c7e89d4","Type":"ContainerDied","Data":"5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd"} Oct 30 05:53:29.952335 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.952313 5072 scope.go:122] "RemoveContainer" containerID="5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd" Oct 30 05:53:29.955190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.954349 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:29.955190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.954440 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:29.955190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.954516 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:29.955190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.954964 5072 generic.go:356] "Generic (PLEG): container finished" podID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerID="d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777" exitCode=0 Oct 30 05:53:29.955190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.955033 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" event={"ID":"cf2b7891-fa63-4011-beaa-aa8692e24cd7","Type":"ContainerDied","Data":"d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777"} Oct 30 05:53:29.955537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.955448 5072 scope.go:122] "RemoveContainer" containerID="d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777" Oct 30 05:53:29.957392 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.957365 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:29.957534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.957473 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:29.957534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:29.957518 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:30.794165 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.794126 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss"] Oct 30 05:53:30.802101 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.800963 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.827986 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:53:30.827938 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:53:30.835498 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.833174 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss"] Oct 30 05:53:30.842860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.842824 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.843004 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.842882 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d474a0-f643-48e6-b1e3-f97070a90298-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.843004 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.842920 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.843116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.843105 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.844537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.843229 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjhh\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-kube-api-access-4rjhh\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.844537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.843286 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.844537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.843313 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.844537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.843369 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d474a0-f643-48e6-b1e3-f97070a90298-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.844537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.843531 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.945139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.945091 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.945394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.945199 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjhh\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-kube-api-access-4rjhh\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.945394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.945253 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.945394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.945278 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.945394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.945326 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d474a0-f643-48e6-b1e3-f97070a90298-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.945734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.945446 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.945734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.945576 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.945734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.945609 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d474a0-f643-48e6-b1e3-f97070a90298-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.945734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.945646 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.946282 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.946150 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.947381 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.947053 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.948283 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.948241 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.948836 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.948809 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d474a0-f643-48e6-b1e3-f97070a90298-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.949121 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.949100 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.950385 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.950360 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d474a0-f643-48e6-b1e3-f97070a90298-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.952127 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.952102 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.970976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.970944 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.973810 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.971884 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" event={"ID":"cf2b7891-fa63-4011-beaa-aa8692e24cd7","Type":"ContainerStarted","Data":"5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4"} Oct 30 05:53:30.975796 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.975767 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" event={"ID":"9b4a3b88-8b55-4812-b640-df363c7e89d4","Type":"ContainerStarted","Data":"76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a"} Oct 30 05:53:30.981284 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.979560 5072 generic.go:356] "Generic (PLEG): container finished" podID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerID="fcb861e348d08f9bc45bacd3b6e51b0a9388bb2916acf9d6de2440c7e4b5b185" exitCode=0 Oct 30 05:53:30.981284 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.979633 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" event={"ID":"45652c65-84b3-4463-92fe-9bb9a8e230f7","Type":"ContainerDied","Data":"fcb861e348d08f9bc45bacd3b6e51b0a9388bb2916acf9d6de2440c7e4b5b185"} Oct 30 05:53:30.982502 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.982110 5072 scope.go:122] "RemoveContainer" containerID="fcb861e348d08f9bc45bacd3b6e51b0a9388bb2916acf9d6de2440c7e4b5b185" Oct 30 05:53:30.983273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.983245 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjhh\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-kube-api-access-4rjhh\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:30.984849 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.984806 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:30.984961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.984882 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:30.984961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:30.984913 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:31.175900 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.175634 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:31.474223 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.463917 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:53:31.474223 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.473082 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:31.474223 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.473136 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:31.496039 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.495835 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:53:31.496039 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.495939 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:31.496039 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.495978 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:31.988553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.988521 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss"] Oct 30 05:53:31.990826 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:53:31.990802 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d474a0_f643_48e6_b1e3_f97070a90298.slice/crio-d1748d05c27e37dbcf5a62b2dfde55637f7115779cdca50a3368fb07482ec95e WatchSource:0}: Error finding container d1748d05c27e37dbcf5a62b2dfde55637f7115779cdca50a3368fb07482ec95e: Status 404 returned error can't find the container with id d1748d05c27e37dbcf5a62b2dfde55637f7115779cdca50a3368fb07482ec95e Oct 30 05:53:31.995147 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.995107 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:31.995262 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.995174 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:31.995262 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.995209 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:31.997695 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:31.997657 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" event={"ID":"45652c65-84b3-4463-92fe-9bb9a8e230f7","Type":"ContainerStarted","Data":"693efd0944f009f952915003933de8b80ced4e44528e394b357eab150dc5f7a3"} Oct 30 05:53:32.149719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:32.139104 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:53:32.149719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:32.139231 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:32.149719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:32.139272 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:32.534194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:32.533079 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:32.534194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:32.533136 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:33.072505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.064098 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:33.072505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.064167 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:33.151484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.140608 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:33.151484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.140682 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:33.581492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.553832 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:53:33.610483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.607945 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:33.610483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.608018 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:33.643696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.630023 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:33.643696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.630092 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:33.846537 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:53:33.821530 5072 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.292s" Oct 30 05:53:33.897691 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.897655 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8"] Oct 30 05:53:33.964521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.916642 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:34.002481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:33.998617 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:34.268712 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.268658 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:34.338170 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.338124 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:34.338658 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.332492 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podStartSLOduration=4.332453002 podStartE2EDuration="4.332453002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:34.297271542 +0000 UTC m=+1373.900960800" watchObservedRunningTime="2026-10-30 05:53:34.332453002 +0000 UTC m=+1373.936142265" Oct 30 05:53:34.396194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.269384 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:34.399188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.399145 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:34.511484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.475628 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:34.511484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.475703 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:34.511484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.505235 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:34.511484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.505999 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" event={"ID":"25d474a0-f643-48e6-b1e3-f97070a90298","Type":"ContainerStarted","Data":"f637cd64b1caf3ffc98fe178cad824370ec08a99e59dfd43b1a8145125fc073a"} Oct 30 05:53:34.511484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.507321 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" event={"ID":"25d474a0-f643-48e6-b1e3-f97070a90298","Type":"ContainerStarted","Data":"d1748d05c27e37dbcf5a62b2dfde55637f7115779cdca50a3368fb07482ec95e"} Oct 30 05:53:34.511484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.507555 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8"] Oct 30 05:53:34.511484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.507616 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:53:34.511484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.508088 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:34.511484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.508152 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:34.517318 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.517293 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:34.562996 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.562901 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-b4ca14-serving-cert\"" Oct 30 05:53:34.948486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.922657 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:53:34.948486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.928647 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:53:34.948486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.935570 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6cw6\" (UniqueName: \"kubernetes.io/projected/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-kube-api-access-p6cw6\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:34.948486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.935646 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:34.948486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.935685 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:34.961281 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.961243 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:34.977649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:34.976496 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:53:35.150911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.150799 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 05:53:35.223640 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.215864 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6cw6\" (UniqueName: \"kubernetes.io/projected/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-kube-api-access-p6cw6\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:35.245496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.243325 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:35.245496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.243371 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:35.245496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.243732 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:35.245496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.224552 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:53:35.275948 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.263594 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:35.275948 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.263659 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:35.358484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.325281 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:35.358484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.325339 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:35.392558 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.363473 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:35.550490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.548789 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:35.639605 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.639554 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:35.668485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.656436 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:35.668485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.657725 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:35.668485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.657772 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:35.668485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.658257 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:35.668485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.658333 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:35.786484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.786083 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6cw6\" (UniqueName: \"kubernetes.io/projected/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-kube-api-access-p6cw6\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:35.828761 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.808148 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:35.934815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:35.934772 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:36.027968 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.027934 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:36.156921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.153725 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:36.161743 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.161711 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:36.161984 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.161956 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:36.191509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.188616 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:36.191509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.188672 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:36.466571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.445259 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:36.466571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.445331 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:36.527119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.525718 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:36.527119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.525776 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:36.564229 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.549309 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:53:36.567638 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.567608 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:53:36.678520 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.673678 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:36.678520 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.674385 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:53:36.904523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.897887 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:36.960484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:36.959834 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:53:37.150877 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.146887 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:37.151419 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.151383 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:37.180564 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.171123 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:37.208323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.202801 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:37.208323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.202867 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:37.456414 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.456318 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:37.456704 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.456671 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:37.468447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.468393 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:53:37.509487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.494521 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:37.509487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.494584 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:37.602524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.596662 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:37.706399 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.693552 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:37.911985 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.910590 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:37.924495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:37.912957 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:53:38.022200 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.020607 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Oct 30 05:53:38.022200 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.021645 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:53:38.215484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.190506 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:38.215767 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.203910 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:38.216271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.215733 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:38.226616 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.226582 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:38.226882 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.226850 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:38.457495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.451107 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:38.457495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.451165 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:38.532161 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.532122 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:38.532804 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.532774 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:38.598917 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.598861 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:38.931488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:38.922692 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:39.022754 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.022708 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:39.099254 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.099218 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:53:39.099546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.099526 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8"] Oct 30 05:53:39.141490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.136603 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:39.141490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.136680 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:39.141490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.138763 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:39.274527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.273701 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:39.278482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.276337 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:39.278482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.276402 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:39.495326 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.467518 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:39.495326 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.467572 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:39.612602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.612536 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:39.669488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.638623 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:39.767524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.767485 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:39.772526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.767777 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:39.777511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:39.772976 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:53:40.055609 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.055558 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:40.166542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.140741 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:40.166542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.166288 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:40.200837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.200804 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:40.201107 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.201080 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:40.298480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.275061 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 05:53:40.461487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.457600 5072 patch_prober.go:29] interesting pod/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 05:53:40.461487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.457662 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 05:53:40.461487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.458716 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" event={"ID":"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f","Type":"ContainerStarted","Data":"699cc5ece9b9d370be173f4b1588f6e562e0eeb0e48094fec2550e1e6749ac92"} Oct 30 05:53:40.507499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.501777 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 05:53:40.507499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.501842 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 05:53:40.696581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.682069 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:40.714528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.713119 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:53:40.835978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.835945 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:53:40.960537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:40.926202 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:53:41.196795 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.196758 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:41.197117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.197069 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:41.290327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.204551 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:41.311486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.211862 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:41.311486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.308652 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:41.311486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.275651 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:41.311486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.232669 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:41.538496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.537250 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:53:41.539084 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.539062 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:53:41.546527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.546286 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:53:41.569514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.556160 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 05:53:41.753482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.718800 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:53:41.753482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.719281 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:41.789719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:41.769888 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:53:42.010628 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.010577 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:42.011512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.011489 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:53:42.089212 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.089085 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" event={"ID":"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f","Type":"ContainerStarted","Data":"51bb7cb0d62866387eeb95d5f24e4aaad8bdd3b096b9f1330124d3e82b372d6e"} Oct 30 05:53:42.090359 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.089616 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:42.092873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.092814 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:53:42.170643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.170605 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:42.170910 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.170883 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:42.222740 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.222700 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:42.222988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.222960 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:42.366392 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.366353 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:53:42.525482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.520230 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" podStartSLOduration=9.6663028 podStartE2EDuration="10.520209331s" totalImagesPullingTime="853.906531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:42.252580899 +0000 UTC m=+1381.856270157" watchObservedRunningTime="2026-10-30 05:53:42.520209331 +0000 UTC m=+1382.123898588" Oct 30 05:53:42.720945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.720839 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:42.722474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:42.722420 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:53:43.012543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.011398 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:53:43.238439 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.238392 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 05:53:43.261095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.249892 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 05:53:43.261583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.261425 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 05:53:43.298542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.294505 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:43.505433 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.505391 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:43.505701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.446841 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.505851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.401553 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:53:43.955540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.955490 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:53:43.956046 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:43.956022 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:53:44.260995 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:44.260889 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:44.262035 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:44.262009 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:53:44.277692 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:44.270210 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 05:53:44.277692 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:44.270286 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 05:53:44.392214 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:44.392183 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:53:44.450485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:44.423721 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:44.511630 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:44.507173 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:53:45.169537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.138239 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:45.263714 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.263671 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:45.308499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.308423 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:53:45.377197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.377169 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:45.377527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.377508 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:53:45.377717 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.377694 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 05:53:45.377918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.377886 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:53:45.381179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.381162 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:45.414658 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.414627 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:53:45.446115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.446023 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:53:45.446815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:45.446795 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:53:48.861934 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:48.861886 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd"] Oct 30 05:53:48.890827 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:48.890793 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.006875 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.006797 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.007206 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.007179 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-256lg\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-kube-api-access-256lg\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.007376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.007361 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.007509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.007495 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.007771 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.007750 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.007922 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.007901 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.008042 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.008026 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.008176 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.008160 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98048f3c-0f13-4c88-b03a-49a0d055e156-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.008349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.008328 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.067579 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.067546 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd"] Oct 30 05:53:49.113661 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.113557 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.113661 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.113614 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.113661 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.113656 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98048f3c-0f13-4c88-b03a-49a0d055e156-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.113946 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.113730 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.113946 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.113938 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.114036 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.113981 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256lg\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-kube-api-access-256lg\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.114091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.114053 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.114091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.114084 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.114219 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.114200 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.114692 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.114660 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.114889 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.114867 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.115077 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.115055 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.115157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.115123 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98048f3c-0f13-4c88-b03a-49a0d055e156-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.115370 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.115349 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.117150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.117127 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.117754 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.117730 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.138045 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.138012 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-256lg\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-kube-api-access-256lg\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.138192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.138155 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.382236 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.382144 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:49.856213 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.856184 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd"] Oct 30 05:53:49.857969 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:53:49.857944 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98048f3c_0f13_4c88_b03a_49a0d055e156.slice/crio-c402cd710c1f5a5990a375b10d127bcd35bb0c9b46ac2ad3f00d8c2be28736cf WatchSource:0}: Error finding container c402cd710c1f5a5990a375b10d127bcd35bb0c9b46ac2ad3f00d8c2be28736cf: Status 404 returned error can't find the container with id c402cd710c1f5a5990a375b10d127bcd35bb0c9b46ac2ad3f00d8c2be28736cf Oct 30 05:53:49.861068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.861030 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:49.861170 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.861113 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:49.861170 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:49.861158 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:53:50.368417 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:50.368382 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" event={"ID":"98048f3c-0f13-4c88-b03a-49a0d055e156","Type":"ContainerStarted","Data":"60fab8e1061bdc157d798e5b2832a17b701bf8c917e7da167cefc9792a3b0df6"} Oct 30 05:53:50.369068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:50.368445 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" event={"ID":"98048f3c-0f13-4c88-b03a-49a0d055e156","Type":"ContainerStarted","Data":"c402cd710c1f5a5990a375b10d127bcd35bb0c9b46ac2ad3f00d8c2be28736cf"} Oct 30 05:53:50.382549 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:50.382519 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:50.383979 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:50.383957 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:53:50.384074 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:50.384004 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 05:53:50.432510 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:50.432423 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" podStartSLOduration=2.432408351 podStartE2EDuration="2.432408351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:50.430540341 +0000 UTC m=+1390.034229599" watchObservedRunningTime="2026-10-30 05:53:50.432408351 +0000 UTC m=+1390.036097607" Oct 30 05:53:51.382960 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:51.382927 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:53:51.383333 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:51.382998 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 05:53:52.424486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:52.420371 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:53:52.424486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:52.420429 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 05:53:53.382873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:53.382837 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:53:53.383055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:53.382901 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 05:53:54.386482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:54.383872 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:53:54.386482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:54.383938 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 05:53:55.383439 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:55.383398 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:53:55.383631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:55.383478 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 05:53:56.383612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:56.383572 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:53:56.384117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:56.383638 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 05:53:57.941071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:57.941035 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:57.942130 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:57.942104 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:53:57.947314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:53:57.947193 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:54:00.715138 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:54:00.715092 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:54:10.137826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.137786 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984"] Oct 30 05:54:10.153750 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.153724 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.175904 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.175878 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fd1c5d-openshift-default-dockercfg-sr6dp\"" Oct 30 05:54:10.178434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.178409 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984"] Oct 30 05:54:10.307711 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.307674 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-certs\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.307887 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.307803 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-credential-socket\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.307887 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.307823 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-token\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.307962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.307914 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-podinfo\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.308005 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.307963 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-socket\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.308080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.308061 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-envoy\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.308138 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.308124 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98r2\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-kube-api-access-w98r2\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.308266 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.308253 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e8eee0-9ec5-401b-9274-9a6319249b65-istiod-ca-cert\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.308383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.308372 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-data\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.409895 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.409801 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-data\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.410772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.410748 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-certs\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.411348 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.411323 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-credential-socket\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.419933 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.419912 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-token\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.420521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.420497 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-podinfo\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.421158 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.421135 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-socket\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.421702 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.421683 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-envoy\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.437372 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.437343 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98r2\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-kube-api-access-w98r2\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.437745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.437722 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e8eee0-9ec5-401b-9274-9a6319249b65-istiod-ca-cert\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.438001 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.421546 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-socket\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.438112 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.411176 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-certs\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.438196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.437078 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-envoy\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.475486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.410580 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-data\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.475486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.419855 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-credential-socket\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.475486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.451238 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e8eee0-9ec5-401b-9274-9a6319249b65-istiod-ca-cert\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.529119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.529083 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-podinfo\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.568560 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.565172 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-token\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.590604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.585146 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98r2\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-kube-api-access-w98r2\") pod \"e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:10.795672 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:10.795565 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:11.480478 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:11.480398 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984"] Oct 30 05:54:11.521419 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:54:11.521350 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e8eee0_9ec5_401b_9274_9a6319249b65.slice/crio-0813c4fb98798c179504021bf40693fb97eb9c37dd6928bd8a648612362ef99f WatchSource:0}: Error finding container 0813c4fb98798c179504021bf40693fb97eb9c37dd6928bd8a648612362ef99f: Status 404 returned error can't find the container with id 0813c4fb98798c179504021bf40693fb97eb9c37dd6928bd8a648612362ef99f Oct 30 05:54:11.525390 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:11.525356 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:54:11.525522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:11.525446 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:54:11.525522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:11.525513 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:54:12.050859 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:12.050809 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" event={"ID":"16e8eee0-9ec5-401b-9274-9a6319249b65","Type":"ContainerStarted","Data":"6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0"} Oct 30 05:54:12.051030 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:12.050879 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" event={"ID":"16e8eee0-9ec5-401b-9274-9a6319249b65","Type":"ContainerStarted","Data":"0813c4fb98798c179504021bf40693fb97eb9c37dd6928bd8a648612362ef99f"} Oct 30 05:54:12.109877 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:12.109814 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podStartSLOduration=2.1097951 podStartE2EDuration="2.1097951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:12.108833354 +0000 UTC m=+1411.712522638" watchObservedRunningTime="2026-10-30 05:54:12.1097951 +0000 UTC m=+1411.713484359" Oct 30 05:54:12.798420 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:12.798378 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:12.804521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:12.804497 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:13.203408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:13.203377 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:13.251867 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:13.251836 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:54:16.738076 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.738037 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk"] Oct 30 05:54:16.747450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.747420 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.767947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.767918 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-df544d-openshift-default-dockercfg-6dwdl\"" Oct 30 05:54:16.808631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.808596 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk"] Oct 30 05:54:16.896486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.875645 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd3999c-39f6-4edf-9b57-71492639891d-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.896486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.875786 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.896486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.875898 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.896486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.875946 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.896486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.876008 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd3999c-39f6-4edf-9b57-71492639891d-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.896486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.876196 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.896486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.876335 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.896486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.876360 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.896486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.876389 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh7ww\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-kube-api-access-rh7ww\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.997556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.987792 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.997556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.987938 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.997556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.987970 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.997556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.987998 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh7ww\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-kube-api-access-rh7ww\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.997556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.988193 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd3999c-39f6-4edf-9b57-71492639891d-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.997556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.988252 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.997556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.988307 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.997556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.988347 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:16.997556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:16.988401 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd3999c-39f6-4edf-9b57-71492639891d-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.004005 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.003773 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.004005 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.003803 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.004135 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.004063 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.063524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.004691 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.063524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.004742 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd3999c-39f6-4edf-9b57-71492639891d-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.216526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.172758 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd3999c-39f6-4edf-9b57-71492639891d-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.216526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.182375 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh7ww\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-kube-api-access-rh7ww\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.216526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.196382 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.545484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.522059 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8jgfk\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:17.670427 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:17.670396 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:18.445538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:18.438767 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk"] Oct 30 05:54:18.460096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:18.459975 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:54:18.460534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:18.460505 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:54:18.460780 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:18.460758 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:54:18.473493 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:18.470540 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" event={"ID":"5fd3999c-39f6-4edf-9b57-71492639891d","Type":"ContainerStarted","Data":"69ccd6c420df541d0b59408a84b3fbb94cc5740360500d7f6f5074823e391f83"} Oct 30 05:54:19.568482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:19.567095 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" event={"ID":"5fd3999c-39f6-4edf-9b57-71492639891d","Type":"ContainerStarted","Data":"611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f"} Oct 30 05:54:19.663315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:19.662600 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podStartSLOduration=3.662578632 podStartE2EDuration="3.662578632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:19.626896889 +0000 UTC m=+1419.230586163" watchObservedRunningTime="2026-10-30 05:54:19.662578632 +0000 UTC m=+1419.266267886" Oct 30 05:54:19.673099 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:19.670815 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:20.679900 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:20.679864 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:21.022660 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.022566 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8"] Oct 30 05:54:21.029637 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.029599 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.067255 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.067232 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e94bd8-openshift-default-dockercfg-nttsj\"" Oct 30 05:54:21.075586 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.075550 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8"] Oct 30 05:54:21.134515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.134473 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.134718 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.134598 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.135203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.134743 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.135203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.134778 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.135203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.134872 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.135203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.135096 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.135203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.135145 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.135203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.135170 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqddh\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-kube-api-access-fqddh\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.135639 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.135316 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.237143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.237107 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.237353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.237183 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.237353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.237213 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqddh\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-kube-api-access-fqddh\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.237353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.237274 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.237353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.237301 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.237633 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.237357 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.237633 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.237523 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.237633 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.237558 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.237859 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.237839 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.238583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.238560 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.238734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.238648 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.239102 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.238953 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.239102 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.239061 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.239259 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.239234 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.241639 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.241619 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.241905 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.241882 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.263320 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.263294 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.263835 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.263815 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqddh\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-kube-api-access-fqddh\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:21.394397 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:21.394361 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:22.015482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:22.006674 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:22.034297 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:22.016387 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:54:22.440254 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:22.429549 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.734562 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:54:22.734532 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b46fb9_e9b9_4cc8_925f_430c5fa73259.slice/crio-e482d2f566ecc94ab03092a05d4245dda84683fc08063cefa20553153954c639 WatchSource:0}: Error finding container e482d2f566ecc94ab03092a05d4245dda84683fc08063cefa20553153954c639: Status 404 returned error can't find the container with id e482d2f566ecc94ab03092a05d4245dda84683fc08063cefa20553153954c639 Oct 30 05:54:22.767972 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:22.764500 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:54:22.775886 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:22.775845 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:54:22.780410 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:22.780376 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:54:22.781163 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:22.780293 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8"] Oct 30 05:54:23.638486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:23.623003 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" event={"ID":"f6b46fb9-e9b9-4cc8-925f-430c5fa73259","Type":"ContainerStarted","Data":"e482d2f566ecc94ab03092a05d4245dda84683fc08063cefa20553153954c639"} Oct 30 05:54:23.775669 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:23.775610 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:23.924488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:23.923863 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:54:23.924488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:23.923923 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:24.623497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:24.623440 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" event={"ID":"f6b46fb9-e9b9-4cc8-925f-430c5fa73259","Type":"ContainerStarted","Data":"e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344"} Oct 30 05:54:24.734495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:24.725167 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podStartSLOduration=4.725147273 podStartE2EDuration="4.725147273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:24.715583777 +0000 UTC m=+1424.319273075" watchObservedRunningTime="2026-10-30 05:54:24.725147273 +0000 UTC m=+1424.328836531" Oct 30 05:54:25.406049 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:25.406017 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:25.409541 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:25.409510 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:54:25.409819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:25.409771 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 05:54:26.400843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:26.400812 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:26.655248 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:26.652429 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:26.655248 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:26.653423 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:54:30.759513 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:54:30.759435 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:54:53.484813 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:53.484775 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss"] Oct 30 05:54:53.485394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:53.485072 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" podUID="46b909d1-426d-4d37-8324-4b01aa536802" containerName="maas-api" containerID="cri-o://808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2" gracePeriod=30 Oct 30 05:54:54.313910 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.313876 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:54:54.432295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.432032 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-metrics-certs\") pod \"46b909d1-426d-4d37-8324-4b01aa536802\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " Oct 30 05:54:54.448838 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.448806 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-maas-api-tls\") pod \"46b909d1-426d-4d37-8324-4b01aa536802\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " Oct 30 05:54:54.449133 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.449115 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b909d1-426d-4d37-8324-4b01aa536802-kube-api-access-bsg8c\" (UniqueName: \"kubernetes.io/projected/46b909d1-426d-4d37-8324-4b01aa536802-kube-api-access-bsg8c\") pod \"46b909d1-426d-4d37-8324-4b01aa536802\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " Oct 30 05:54:54.452267 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.452237 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b909d1-426d-4d37-8324-4b01aa536802-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b909d1-426d-4d37-8324-4b01aa536802-openshift-service-ca\") pod \"46b909d1-426d-4d37-8324-4b01aa536802\" (UID: \"46b909d1-426d-4d37-8324-4b01aa536802\") " Oct 30 05:54:54.554002 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.553944 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b909d1-426d-4d37-8324-4b01aa536802-openshift-service-ca" pod "46b909d1-426d-4d37-8324-4b01aa536802" (UID: "46b909d1-426d-4d37-8324-4b01aa536802"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:54.680335 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.680272 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46b909d1-426d-4d37-8324-4b01aa536802-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:54:54.687230 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.687042 5072 generic.go:356] "Generic (PLEG): container finished" podID="46b909d1-426d-4d37-8324-4b01aa536802" containerID="808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2" exitCode=0 Oct 30 05:54:54.687775 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.687734 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" Oct 30 05:54:54.835591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.789202 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b909d1-426d-4d37-8324-4b01aa536802-kube-api-access-bsg8c" pod "46b909d1-426d-4d37-8324-4b01aa536802" (UID: "46b909d1-426d-4d37-8324-4b01aa536802"). InnerVolumeSpecName "kube-api-access-bsg8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:54.835591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.827259 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-maas-api-tls" pod "46b909d1-426d-4d37-8324-4b01aa536802" (UID: "46b909d1-426d-4d37-8324-4b01aa536802"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:54.835591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.833356 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-metrics-certs" pod "46b909d1-426d-4d37-8324-4b01aa536802" (UID: "46b909d1-426d-4d37-8324-4b01aa536802"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:54.835591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.822383 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" event={"ID":"46b909d1-426d-4d37-8324-4b01aa536802","Type":"ContainerDied","Data":"808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2"} Oct 30 05:54:54.835591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.834328 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss" event={"ID":"46b909d1-426d-4d37-8324-4b01aa536802","Type":"ContainerDied","Data":"4d8d29e0bfe5f63de7da10107a6dc915cdb3e31671d4a1c7a27b103d1fed3ea8"} Oct 30 05:54:54.835591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.834380 5072 scope.go:122] "RemoveContainer" containerID="808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2" Oct 30 05:54:54.845858 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.844628 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z"] Oct 30 05:54:54.849500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.846972 5072 cpu_manager.go:414] "containerMap: removing container" podUID="46b909d1-426d-4d37-8324-4b01aa536802" containerName="maas-api" Oct 30 05:54:54.849500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.846995 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b909d1-426d-4d37-8324-4b01aa536802" containerName="maas-api" Oct 30 05:54:54.849500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.847861 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b909d1-426d-4d37-8324-4b01aa536802" containerName="maas-api" Oct 30 05:54:54.872190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.872155 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z"] Oct 30 05:54:54.872837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.872560 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:54.880615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.876449 5072 scope.go:122] "RemoveContainer" containerID="808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2" Oct 30 05:54:54.880615 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:54:54.876870 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2\": container with ID starting with 808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2 not found: ID does not exist" containerID="808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2" Oct 30 05:54:54.880615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.876903 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2"} err="failed to get container status \"808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2\": rpc error: code = NotFound desc = could not find container \"808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2\": container with ID starting with 808b5e9e01725ec06565cdc9dc62293cb09afd4970d279a3f4bf02a829bf7aa2 not found: ID does not exist" Oct 30 05:54:54.892270 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.892241 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e94bd8-serving-cert\"" Oct 30 05:54:54.931605 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.931572 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-metrics-certs\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:54.931802 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.931782 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e504e452-c65d-4e2a-8729-505c764d7f6b-openshift-service-ca\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:54.931973 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.931951 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpp4p\" (UniqueName: \"kubernetes.io/projected/e504e452-c65d-4e2a-8729-505c764d7f6b-kube-api-access-wpp4p\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:54.932093 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.932079 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:54.932263 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.932246 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:54:54.932354 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.932266 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46b909d1-426d-4d37-8324-4b01aa536802-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:54:54.932354 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:54.932280 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsg8c\" (UniqueName: \"kubernetes.io/projected/46b909d1-426d-4d37-8324-4b01aa536802-kube-api-access-bsg8c\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:54:55.036537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.035998 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:55.036537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.036232 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-metrics-certs\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:55.036537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.036288 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e504e452-c65d-4e2a-8729-505c764d7f6b-openshift-service-ca\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:55.036537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.036517 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpp4p\" (UniqueName: \"kubernetes.io/projected/e504e452-c65d-4e2a-8729-505c764d7f6b-kube-api-access-wpp4p\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:55.052572 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:54:55.037099 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-serving-cert: secret "maas-api-e2e-api-a-e94bd8-serving-cert" not found Oct 30 05:54:55.052572 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:54:55.037183 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls podName:e504e452-c65d-4e2a-8729-505c764d7f6b nodeName:}" failed. No retries permitted until 2026-10-30 05:54:55.537161643 +0000 UTC m=+1455.140850895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls") pod "maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" (UID: "e504e452-c65d-4e2a-8729-505c764d7f6b") : secret "maas-api-e2e-api-a-e94bd8-serving-cert" not found Oct 30 05:54:55.052572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.038511 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e504e452-c65d-4e2a-8729-505c764d7f6b-openshift-service-ca\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:55.052572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.046878 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-metrics-certs\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:55.072950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.072913 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpp4p\" (UniqueName: \"kubernetes.io/projected/e504e452-c65d-4e2a-8729-505c764d7f6b-kube-api-access-wpp4p\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:55.243941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.243912 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss"] Oct 30 05:54:55.250598 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.250573 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8bab43d7-7ffd5bf568-4h9ss"] Oct 30 05:54:55.544666 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.544630 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:55.547398 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.547372 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls\") pod \"maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:55.811883 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:55.811772 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:56.176781 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:56.176747 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z"] Oct 30 05:54:56.177092 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:54:56.177065 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode504e452_c65d_4e2a_8729_505c764d7f6b.slice/crio-5dfe65f7d113c32a90dfff78e4c33ce3fe1bf8e5f858870c1242e841d65ef28d WatchSource:0}: Error finding container 5dfe65f7d113c32a90dfff78e4c33ce3fe1bf8e5f858870c1242e841d65ef28d: Status 404 returned error can't find the container with id 5dfe65f7d113c32a90dfff78e4c33ce3fe1bf8e5f858870c1242e841d65ef28d Oct 30 05:54:56.536120 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:56.535748 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b909d1-426d-4d37-8324-4b01aa536802" path="/var/lib/kubelet/pods/46b909d1-426d-4d37-8324-4b01aa536802/volumes" Oct 30 05:54:56.730037 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:56.729999 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" event={"ID":"e504e452-c65d-4e2a-8729-505c764d7f6b","Type":"ContainerStarted","Data":"5dfe65f7d113c32a90dfff78e4c33ce3fe1bf8e5f858870c1242e841d65ef28d"} Oct 30 05:54:57.740851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:57.740815 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" event={"ID":"e504e452-c65d-4e2a-8729-505c764d7f6b","Type":"ContainerStarted","Data":"6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338"} Oct 30 05:54:57.741336 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:57.740883 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:54:57.784043 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:54:57.783994 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" podStartSLOduration=3.360169914 podStartE2EDuration="3.7839808s" totalImagesPullingTime="423.810886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:57.781852386 +0000 UTC m=+1457.385541670" watchObservedRunningTime="2026-10-30 05:54:57.7839808 +0000 UTC m=+1457.387670056" Oct 30 05:55:00.715188 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:00.715147 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:55:02.659358 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:02.659272 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs"] Oct 30 05:55:02.660044 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:02.659545 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" containerID="cri-o://5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4" gracePeriod=30 Oct 30 05:55:03.665870 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:03.665836 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp"] Oct 30 05:55:03.666306 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:03.666114 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" containerID="cri-o://555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1" gracePeriod=30 Oct 30 05:55:03.880123 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:03.879638 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:55:07.976665 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:07.976638 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:55:08.035476 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.035415 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-token\") pod \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " Oct 30 05:55:08.035673 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.035536 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-envoy\") pod \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " Oct 30 05:55:08.035673 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.035574 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-podinfo\") pod \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " Oct 30 05:55:08.036048 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.036019 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-data\") pod \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " Oct 30 05:55:08.036409 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.036107 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-socket\") pod \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " Oct 30 05:55:08.036409 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.036278 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-certs\") pod \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " Oct 30 05:55:08.036409 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.036355 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-credential-socket\") pod \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " Oct 30 05:55:08.036409 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.036400 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-kube-api-access-rpn97\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-kube-api-access-rpn97\") pod \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " Oct 30 05:55:08.036744 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.036410 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-data" pod "cf2b7891-fa63-4011-beaa-aa8692e24cd7" (UID: "cf2b7891-fa63-4011-beaa-aa8692e24cd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:08.036744 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.036637 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istiod-ca-cert\") pod \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\" (UID: \"cf2b7891-fa63-4011-beaa-aa8692e24cd7\") " Oct 30 05:55:08.036744 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.036658 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-certs" pod "cf2b7891-fa63-4011-beaa-aa8692e24cd7" (UID: "cf2b7891-fa63-4011-beaa-aa8692e24cd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:08.036904 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.036835 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-credential-socket" pod "cf2b7891-fa63-4011-beaa-aa8692e24cd7" (UID: "cf2b7891-fa63-4011-beaa-aa8692e24cd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:08.037231 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.037200 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istiod-ca-cert" pod "cf2b7891-fa63-4011-beaa-aa8692e24cd7" (UID: "cf2b7891-fa63-4011-beaa-aa8692e24cd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:08.037681 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.037656 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:08.037781 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.037686 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:08.037781 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.037698 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:08.037781 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.037711 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:08.042988 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.042933 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-token" pod "cf2b7891-fa63-4011-beaa-aa8692e24cd7" (UID: "cf2b7891-fa63-4011-beaa-aa8692e24cd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:08.043689 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.043663 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-envoy" pod "cf2b7891-fa63-4011-beaa-aa8692e24cd7" (UID: "cf2b7891-fa63-4011-beaa-aa8692e24cd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:08.043878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.043828 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-kube-api-access-rpn97" pod "cf2b7891-fa63-4011-beaa-aa8692e24cd7" (UID: "cf2b7891-fa63-4011-beaa-aa8692e24cd7"). InnerVolumeSpecName "kube-api-access-rpn97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:08.044052 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.044028 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-podinfo" pod "cf2b7891-fa63-4011-beaa-aa8692e24cd7" (UID: "cf2b7891-fa63-4011-beaa-aa8692e24cd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:08.044269 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.044238 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-socket" pod "cf2b7891-fa63-4011-beaa-aa8692e24cd7" (UID: "cf2b7891-fa63-4011-beaa-aa8692e24cd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:08.108721 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.108690 5072 generic.go:356] "Generic (PLEG): container finished" podID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerID="5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4" exitCode=0 Oct 30 05:55:08.108912 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.108769 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" event={"ID":"cf2b7891-fa63-4011-beaa-aa8692e24cd7","Type":"ContainerDied","Data":"5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4"} Oct 30 05:55:08.108912 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.108846 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" Oct 30 05:55:08.108912 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.108871 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs" event={"ID":"cf2b7891-fa63-4011-beaa-aa8692e24cd7","Type":"ContainerDied","Data":"678d1aa1be19652b9327fba7a1b71f6fed96bee76c838a0bda80bbb01614bf00"} Oct 30 05:55:08.108912 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.108910 5072 scope.go:122] "RemoveContainer" containerID="5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4" Oct 30 05:55:08.122208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.122181 5072 scope.go:122] "RemoveContainer" containerID="d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777" Oct 30 05:55:08.133018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.132984 5072 scope.go:122] "RemoveContainer" containerID="5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4" Oct 30 05:55:08.133387 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:08.133369 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4\": container with ID starting with 5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4 not found: ID does not exist" containerID="5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4" Oct 30 05:55:08.133511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.133399 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4"} err="failed to get container status \"5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4\": rpc error: code = NotFound desc = could not find container \"5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4\": container with ID starting with 5c9f6f3a5abcd0d32f582720a4830eecd8c46b1a312254295e38c286392e5bb4 not found: ID does not exist" Oct 30 05:55:08.133511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.133417 5072 scope.go:122] "RemoveContainer" containerID="d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777" Oct 30 05:55:08.133710 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:08.133690 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777\": container with ID starting with d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777 not found: ID does not exist" containerID="d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777" Oct 30 05:55:08.133769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.133719 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777"} err="failed to get container status \"d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777\": rpc error: code = NotFound desc = could not find container \"d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777\": container with ID starting with d30a276b6a97a354752f0bae6ce65e45da3479a5284b45771947e9e7e4c65777 not found: ID does not exist" Oct 30 05:55:08.139854 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.139831 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpn97\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-kube-api-access-rpn97\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:08.139854 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.139856 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:08.139971 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.139866 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:08.139971 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.139874 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b7891-fa63-4011-beaa-aa8692e24cd7-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:08.139971 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.139881 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b7891-fa63-4011-beaa-aa8692e24cd7-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:08.165398 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.165367 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs"] Oct 30 05:55:08.187420 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.187345 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8bab43d7-openshift-default-5d8b5d5fc-428vs"] Oct 30 05:55:08.523690 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.523606 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" path="/var/lib/kubelet/pods/cf2b7891-fa63-4011-beaa-aa8692e24cd7/volumes" Oct 30 05:55:08.981826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:08.981801 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:55:09.057591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.057504 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-kube-api-access-qzwt6\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-kube-api-access-qzwt6\") pod \"012b25d5-4f64-439a-a6cd-6ed87e029374\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " Oct 30 05:55:09.057784 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.057621 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-certs\") pod \"012b25d5-4f64-439a-a6cd-6ed87e029374\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " Oct 30 05:55:09.057784 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.057680 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-token\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-token\") pod \"012b25d5-4f64-439a-a6cd-6ed87e029374\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " Oct 30 05:55:09.057784 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.057766 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-credential-socket\") pod \"012b25d5-4f64-439a-a6cd-6ed87e029374\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " Oct 30 05:55:09.057968 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.057893 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-data\") pod \"012b25d5-4f64-439a-a6cd-6ed87e029374\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " Oct 30 05:55:09.057968 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.057958 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-envoy\") pod \"012b25d5-4f64-439a-a6cd-6ed87e029374\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " Oct 30 05:55:09.058104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.058086 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-socket\") pod \"012b25d5-4f64-439a-a6cd-6ed87e029374\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " Oct 30 05:55:09.058172 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.058137 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-podinfo\") pod \"012b25d5-4f64-439a-a6cd-6ed87e029374\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " Oct 30 05:55:09.058232 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.058172 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012b25d5-4f64-439a-a6cd-6ed87e029374-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012b25d5-4f64-439a-a6cd-6ed87e029374-istiod-ca-cert\") pod \"012b25d5-4f64-439a-a6cd-6ed87e029374\" (UID: \"012b25d5-4f64-439a-a6cd-6ed87e029374\") " Oct 30 05:55:09.058373 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.058350 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-socket" pod "012b25d5-4f64-439a-a6cd-6ed87e029374" (UID: "012b25d5-4f64-439a-a6cd-6ed87e029374"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:09.058713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.058689 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-credential-socket" pod "012b25d5-4f64-439a-a6cd-6ed87e029374" (UID: "012b25d5-4f64-439a-a6cd-6ed87e029374"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:09.058911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.058804 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012b25d5-4f64-439a-a6cd-6ed87e029374-istiod-ca-cert" pod "012b25d5-4f64-439a-a6cd-6ed87e029374" (UID: "012b25d5-4f64-439a-a6cd-6ed87e029374"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:09.058911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.058880 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-data" pod "012b25d5-4f64-439a-a6cd-6ed87e029374" (UID: "012b25d5-4f64-439a-a6cd-6ed87e029374"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:09.059177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.059115 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:09.059177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.059142 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012b25d5-4f64-439a-a6cd-6ed87e029374-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:09.059177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.059150 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:09.059177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.059159 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:09.059621 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.059594 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-certs" pod "012b25d5-4f64-439a-a6cd-6ed87e029374" (UID: "012b25d5-4f64-439a-a6cd-6ed87e029374"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:09.061250 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.061224 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-podinfo" pod "012b25d5-4f64-439a-a6cd-6ed87e029374" (UID: "012b25d5-4f64-439a-a6cd-6ed87e029374"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:09.061837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.061795 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-kube-api-access-qzwt6" pod "012b25d5-4f64-439a-a6cd-6ed87e029374" (UID: "012b25d5-4f64-439a-a6cd-6ed87e029374"). InnerVolumeSpecName "kube-api-access-qzwt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:09.062978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.062951 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-envoy" pod "012b25d5-4f64-439a-a6cd-6ed87e029374" (UID: "012b25d5-4f64-439a-a6cd-6ed87e029374"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:09.063271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.063244 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-token" pod "012b25d5-4f64-439a-a6cd-6ed87e029374" (UID: "012b25d5-4f64-439a-a6cd-6ed87e029374"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:09.122869 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.122832 5072 generic.go:356] "Generic (PLEG): container finished" podID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerID="555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1" exitCode=0 Oct 30 05:55:09.123038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.122885 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" Oct 30 05:55:09.123038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.122946 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" event={"ID":"012b25d5-4f64-439a-a6cd-6ed87e029374","Type":"ContainerDied","Data":"555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1"} Oct 30 05:55:09.123038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.122988 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp" event={"ID":"012b25d5-4f64-439a-a6cd-6ed87e029374","Type":"ContainerDied","Data":"c1c6aba02c10b38ff5bf378e5b31ffbc7a8172c99f3eb39d69a7052d197933bc"} Oct 30 05:55:09.123038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.123024 5072 scope.go:122] "RemoveContainer" containerID="555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1" Oct 30 05:55:09.138938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.138906 5072 scope.go:122] "RemoveContainer" containerID="555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1" Oct 30 05:55:09.139368 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:09.139333 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1\": container with ID starting with 555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1 not found: ID does not exist" containerID="555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1" Oct 30 05:55:09.139595 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.139371 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1"} err="failed to get container status \"555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1\": rpc error: code = NotFound desc = could not find container \"555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1\": container with ID starting with 555285b797a7a0443588266da189f1cd6934ccdbed660a12455b11694ec37fb1 not found: ID does not exist" Oct 30 05:55:09.160660 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.160629 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:09.160660 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.160658 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:09.160914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.160672 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwt6\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-kube-api-access-qzwt6\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:09.160914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.160686 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012b25d5-4f64-439a-a6cd-6ed87e029374-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:09.160914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.160697 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012b25d5-4f64-439a-a6cd-6ed87e029374-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:09.184080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.184040 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp"] Oct 30 05:55:09.216197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:09.201931 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-rn9wp"] Oct 30 05:55:10.936815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:10.936027 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" path="/var/lib/kubelet/pods/012b25d5-4f64-439a-a6cd-6ed87e029374/volumes" Oct 30 05:55:17.754482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.694159 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8"] Oct 30 05:55:17.754482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.695400 5072 cpu_manager.go:414] "containerMap: removing container" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" Oct 30 05:55:17.754482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.695425 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" Oct 30 05:55:17.754482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.695496 5072 cpu_manager.go:414] "containerMap: removing container" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" Oct 30 05:55:17.754482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.695505 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" Oct 30 05:55:17.754482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.695733 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" Oct 30 05:55:17.754482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.695757 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="012b25d5-4f64-439a-a6cd-6ed87e029374" containerName="istio-proxy" Oct 30 05:55:17.904728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.904652 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:17.949483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.942657 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-530a5e-openshift-default-dockercfg-2cmk9\"" Oct 30 05:55:17.967499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:17.962571 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8"] Oct 30 05:55:18.107537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.091492 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46689523-9454-44e3-8e76-fb8992bb6c1b-istiod-ca-cert\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.107537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.091601 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-certs\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.107537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.091828 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-podinfo\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.107537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.091871 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rl6\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-kube-api-access-k4rl6\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.107537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.091917 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-socket\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.107537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.091984 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-token\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.107537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.092035 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-credential-socket\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.107537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.092100 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-envoy\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.107537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.092240 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-data\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.239849 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.239789 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-podinfo\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.240115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.240099 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rl6\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-kube-api-access-k4rl6\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.240267 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.240253 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-socket\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.240424 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.240409 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-token\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.240585 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.240569 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-credential-socket\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.240739 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.240726 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-envoy\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.240947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.240934 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-data\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.241094 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.241079 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46689523-9454-44e3-8e76-fb8992bb6c1b-istiod-ca-cert\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.241254 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.241241 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-certs\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.242835 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.242810 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-certs\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.243230 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.243211 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-credential-socket\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.262157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.259275 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-data\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.262157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.259886 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-socket\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.262157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.260531 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46689523-9454-44e3-8e76-fb8992bb6c1b-istiod-ca-cert\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.281638 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.281564 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-token\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.302662 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.302571 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-envoy\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.323861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.311539 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-podinfo\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.325478 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.325372 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rl6\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-kube-api-access-k4rl6\") pod \"e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:18.500809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:18.500490 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:19.496555 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:55:19.496516 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46689523_9454_44e3_8e76_fb8992bb6c1b.slice/crio-b07125552c2e0762529e4342f21da3cab86984be12c33e4b4e03ce7345af90f5 WatchSource:0}: Error finding container b07125552c2e0762529e4342f21da3cab86984be12c33e4b4e03ce7345af90f5: Status 404 returned error can't find the container with id b07125552c2e0762529e4342f21da3cab86984be12c33e4b4e03ce7345af90f5 Oct 30 05:55:19.542365 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:19.542252 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:19.542547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:19.542389 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:19.542547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:19.542449 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:19.554542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:19.551243 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8"] Oct 30 05:55:20.640538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:20.634299 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" event={"ID":"46689523-9454-44e3-8e76-fb8992bb6c1b","Type":"ContainerStarted","Data":"43193e033ef400484b2378d8dad6253081ff741d413ec465a67e7e765d3b6806"} Oct 30 05:55:20.640538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:20.634367 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" event={"ID":"46689523-9454-44e3-8e76-fb8992bb6c1b","Type":"ContainerStarted","Data":"b07125552c2e0762529e4342f21da3cab86984be12c33e4b4e03ce7345af90f5"} Oct 30 05:55:20.725486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:20.699794 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" podStartSLOduration=3.699771816 podStartE2EDuration="3.699771816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:20.682351033 +0000 UTC m=+1480.286040303" watchObservedRunningTime="2026-10-30 05:55:20.699771816 +0000 UTC m=+1480.303461074" Oct 30 05:55:21.507327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:21.506681 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:21.542852 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:21.542825 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:21.670101 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:21.670015 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:21.694631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:21.694217 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:55:24.328553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:24.283968 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:24.669676 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:24.669587 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:55:24.700747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:24.700703 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:24.769518 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:24.768176 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:24.958932 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:24.958826 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:55:25.739481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:25.735880 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8862-df1bcc"] Oct 30 05:55:25.739481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:25.737117 5072 cpu_manager.go:414] "containerMap: removing container" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" Oct 30 05:55:25.739481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:25.737141 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" Oct 30 05:55:25.739481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:25.737414 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2b7891-fa63-4011-beaa-aa8692e24cd7" containerName="istio-proxy" Oct 30 05:55:25.781741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:25.777770 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:26.037881 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:26.037663 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bnq\"" Oct 30 05:55:26.038884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:26.038790 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8862-df1bcc"] Oct 30 05:55:26.054224 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:26.051473 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9tlw\" (UniqueName: \"kubernetes.io/projected/be7a5688-4515-4e86-ab0e-2fec7c3f1aef-kube-api-access-m9tlw\") pod \"test-curl-8862-df1bcc\" (UID: \"be7a5688-4515-4e86-ab0e-2fec7c3f1aef\") " pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:26.154116 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:26.154066 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tlw\" (UniqueName: \"kubernetes.io/projected/be7a5688-4515-4e86-ab0e-2fec7c3f1aef-kube-api-access-m9tlw\") pod \"test-curl-8862-df1bcc\" (UID: \"be7a5688-4515-4e86-ab0e-2fec7c3f1aef\") " pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:26.179117 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:26.179084 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9tlw\" (UniqueName: \"kubernetes.io/projected/be7a5688-4515-4e86-ab0e-2fec7c3f1aef-kube-api-access-m9tlw\") pod \"test-curl-8862-df1bcc\" (UID: \"be7a5688-4515-4e86-ab0e-2fec7c3f1aef\") " pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:26.379060 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:26.379021 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:26.874484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:26.874335 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8862-df1bcc"] Oct 30 05:55:27.696482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:27.695805 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:55:27.696482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:27.695865 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:28.204308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:28.204263 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8862-df1bcc" event={"ID":"be7a5688-4515-4e86-ab0e-2fec7c3f1aef","Type":"ContainerStarted","Data":"a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a"} Oct 30 05:55:28.204308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:28.204308 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8862-df1bcc" event={"ID":"be7a5688-4515-4e86-ab0e-2fec7c3f1aef","Type":"ContainerStarted","Data":"a739d80c5934fad10f8979d660ccaa47a73ef11815d94b9dc40e33e56618bef1"} Oct 30 05:55:28.243486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:28.242426 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8862-df1bcc" podStartSLOduration=3.242408325 podStartE2EDuration="3.242408325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:28.241646569 +0000 UTC m=+1487.845335827" watchObservedRunningTime="2026-10-30 05:55:28.242408325 +0000 UTC m=+1487.846097583" Oct 30 05:55:29.244708 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.244663 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:55:29.245327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.244742 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:29.296063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.296025 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8862-df1bcc"] Oct 30 05:55:29.303065 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.303026 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8862-df1bcc"] Oct 30 05:55:29.572730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.572653 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8"] Oct 30 05:55:29.573541 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.573521 5072 cpu_manager.go:414] "containerMap: removing container" podUID="be7a5688-4515-4e86-ab0e-2fec7c3f1aef" containerName="test-curl-8862-df1bcc" Oct 30 05:55:29.573658 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.573542 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7a5688-4515-4e86-ab0e-2fec7c3f1aef" containerName="test-curl-8862-df1bcc" Oct 30 05:55:29.573799 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.573765 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7a5688-4515-4e86-ab0e-2fec7c3f1aef" containerName="test-curl-8862-df1bcc" Oct 30 05:55:29.658755 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.658618 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8"] Oct 30 05:55:29.658755 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.658745 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.701285 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.690990 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4323f6-openshift-default-dockercfg-9xgct\"" Oct 30 05:55:29.926536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.851089 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-podinfo\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.926536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.851195 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-credential-socket\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.926536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.851318 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqcwc\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-kube-api-access-gqcwc\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.926536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.851380 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-data\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.926536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.851421 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-certs\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.926536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.851532 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-socket\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.926536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.851631 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-envoy\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.926536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.851695 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istiod-ca-cert\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.926536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.851796 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-token\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.957987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.957946 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-token\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.958352 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.958320 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-podinfo\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.969066 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.969033 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-credential-socket\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.974969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.969447 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqcwc\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-kube-api-access-gqcwc\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:29.975363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.975341 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-data\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.001486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.975537 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-certs\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.001892 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.001866 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-socket\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.002231 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.002211 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-envoy\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.002502 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.002450 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istiod-ca-cert\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.003670 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.003648 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istiod-ca-cert\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.016549 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.997280 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-credential-socket\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.016828 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.997849 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-data\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.017072 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.007226 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-socket\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.017191 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:29.998059 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-certs\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.140417 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.101879 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqcwc\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-kube-api-access-gqcwc\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.169143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.169086 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-token\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.194642 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.176799 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-podinfo\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.195162 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.195137 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-envoy\") pod \"e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.261570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.261529 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8862-df1bcc" podUID="be7a5688-4515-4e86-ab0e-2fec7c3f1aef" containerName="test-curl-8862-df1bcc" containerID="cri-o://a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a" gracePeriod=2 Oct 30 05:55:30.288960 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:30.288806 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8862-df1bcc\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="be7a5688-4515-4e86-ab0e-2fec7c3f1aef" pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:30.462821 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:30.462704 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:30.892219 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:30.892179 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8862-df1bcc\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="be7a5688-4515-4e86-ab0e-2fec7c3f1aef" pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:30.987479 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:30.976815 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:55:31.513097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:31.513012 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8"] Oct 30 05:55:31.520223 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:55:31.520175 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee624f7_b2c8_48e1_b001_f2d58bc7a905.slice/crio-54608958c81feac0d57c77b5f22747db2bdbde73b7f5672987bf73a0049afbdd WatchSource:0}: Error finding container 54608958c81feac0d57c77b5f22747db2bdbde73b7f5672987bf73a0049afbdd: Status 404 returned error can't find the container with id 54608958c81feac0d57c77b5f22747db2bdbde73b7f5672987bf73a0049afbdd Oct 30 05:55:31.528010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:31.525642 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:31.528010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:31.525747 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:31.528010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:31.525910 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:32.838187 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:32.838096 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" event={"ID":"8ee624f7-b2c8-48e1-b001-f2d58bc7a905","Type":"ContainerStarted","Data":"b21b202a79fcb11e93b95f25d6f654390b31e35ee448e4cf717622ecb141b845"} Oct 30 05:55:32.838801 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:32.838726 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" event={"ID":"8ee624f7-b2c8-48e1-b001-f2d58bc7a905","Type":"ContainerStarted","Data":"54608958c81feac0d57c77b5f22747db2bdbde73b7f5672987bf73a0049afbdd"} Oct 30 05:55:32.886733 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:32.886655 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podStartSLOduration=3.8866322909999997 podStartE2EDuration="3.886632291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:32.881164821 +0000 UTC m=+1492.484854102" watchObservedRunningTime="2026-10-30 05:55:32.886632291 +0000 UTC m=+1492.490321549" Oct 30 05:55:33.390627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.390598 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:33.453526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.453448 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7a5688-4515-4e86-ab0e-2fec7c3f1aef-kube-api-access-m9tlw\" (UniqueName: \"kubernetes.io/projected/be7a5688-4515-4e86-ab0e-2fec7c3f1aef-kube-api-access-m9tlw\") pod \"be7a5688-4515-4e86-ab0e-2fec7c3f1aef\" (UID: \"be7a5688-4515-4e86-ab0e-2fec7c3f1aef\") " Oct 30 05:55:33.458200 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.458167 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7a5688-4515-4e86-ab0e-2fec7c3f1aef-kube-api-access-m9tlw" pod "be7a5688-4515-4e86-ab0e-2fec7c3f1aef" (UID: "be7a5688-4515-4e86-ab0e-2fec7c3f1aef"). InnerVolumeSpecName "kube-api-access-m9tlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:33.469515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.469343 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:33.491552 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.491514 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:33.557610 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.556763 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9tlw\" (UniqueName: \"kubernetes.io/projected/be7a5688-4515-4e86-ab0e-2fec7c3f1aef-kube-api-access-m9tlw\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:33.926166 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.923617 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8"] Oct 30 05:55:33.926166 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.924262 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" containerID="cri-o://1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55" gracePeriod=30 Oct 30 05:55:33.962197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.948985 5072 generic.go:356] "Generic (PLEG): container finished" podID="be7a5688-4515-4e86-ab0e-2fec7c3f1aef" containerID="a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a" exitCode=137 Oct 30 05:55:33.962197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.950534 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:33.962197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.951419 5072 scope.go:122] "RemoveContainer" containerID="a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a" Oct 30 05:55:33.971763 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.965938 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:33.973335 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:33.973174 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:55:34.021818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:34.021331 5072 scope.go:122] "RemoveContainer" containerID="a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a" Oct 30 05:55:34.021920 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:34.021890 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a\": container with ID starting with a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a not found: ID does not exist" containerID="a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a" Oct 30 05:55:34.021997 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:34.021930 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a"} err="failed to get container status \"a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a\": rpc error: code = NotFound desc = could not find container \"a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a\": container with ID starting with a252ce8edb32794b2e57369e0534bad3bc6afff687077e4f678678c377aef26a not found: ID does not exist" Oct 30 05:55:34.045138 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:34.045084 5072 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8862-df1bcc\" is forbidden: User \"system:node:ip-10-0-138-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-200.ec2.internal' and this object" podUID="be7a5688-4515-4e86-ab0e-2fec7c3f1aef" pod="opendatahub/test-curl-8862-df1bcc" Oct 30 05:55:34.530278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:34.530242 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7a5688-4515-4e86-ab0e-2fec7c3f1aef" path="/var/lib/kubelet/pods/be7a5688-4515-4e86-ab0e-2fec7c3f1aef/volumes" Oct 30 05:55:37.390071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.390035 5072 generic.go:356] "Generic (PLEG): container finished" podID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerID="6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0" exitCode=0 Oct 30 05:55:37.390569 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.390103 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" event={"ID":"16e8eee0-9ec5-401b-9274-9a6319249b65","Type":"ContainerDied","Data":"6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0"} Oct 30 05:55:37.390668 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.390628 5072 scope.go:122] "RemoveContainer" containerID="6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0" Oct 30 05:55:37.392428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.392149 5072 generic.go:356] "Generic (PLEG): container finished" podID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerID="e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344" exitCode=0 Oct 30 05:55:37.392428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.392217 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" event={"ID":"f6b46fb9-e9b9-4cc8-925f-430c5fa73259","Type":"ContainerDied","Data":"e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344"} Oct 30 05:55:37.392773 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.392755 5072 scope.go:122] "RemoveContainer" containerID="e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344" Oct 30 05:55:37.393540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.393386 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:37.393540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.393492 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:37.393670 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.393542 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:37.394894 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.394867 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:37.394971 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.394944 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:37.395022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:37.394991 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:38.095621 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:38.070948 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:55:38.095621 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:38.071007 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:38.465313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:38.465272 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" event={"ID":"16e8eee0-9ec5-401b-9274-9a6319249b65","Type":"ContainerStarted","Data":"00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e"} Oct 30 05:55:38.479486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:38.469301 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" event={"ID":"f6b46fb9-e9b9-4cc8-925f-430c5fa73259","Type":"ContainerStarted","Data":"249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b"} Oct 30 05:55:38.800703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:38.799392 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:55:38.800703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:38.799596 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 05:55:38.800703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:38.799647 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 05:55:39.396325 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.396292 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:55:39.396573 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.396539 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:55:39.396649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.396591 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 05:55:39.493447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.493407 5072 generic.go:356] "Generic (PLEG): container finished" podID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerID="1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55" exitCode=0 Oct 30 05:55:39.493974 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.493502 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" event={"ID":"3cbe543f-fbfb-4950-a940-8b25266ea5be","Type":"ContainerDied","Data":"1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55"} Oct 30 05:55:39.496012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.495991 5072 generic.go:356] "Generic (PLEG): container finished" podID="5fd3999c-39f6-4edf-9b57-71492639891d" containerID="611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f" exitCode=0 Oct 30 05:55:39.497528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.497000 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" event={"ID":"5fd3999c-39f6-4edf-9b57-71492639891d","Type":"ContainerDied","Data":"611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f"} Oct 30 05:55:39.497685 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.497665 5072 scope.go:122] "RemoveContainer" containerID="611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f" Oct 30 05:55:39.500421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.500392 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:39.500622 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.500593 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:39.500742 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.500662 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:39.682661 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.682630 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.703100 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-kube-api-access-s6chn\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-kube-api-access-s6chn\") pod \"3cbe543f-fbfb-4950-a940-8b25266ea5be\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.703234 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-socket\") pod \"3cbe543f-fbfb-4950-a940-8b25266ea5be\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.703264 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-certs\") pod \"3cbe543f-fbfb-4950-a940-8b25266ea5be\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.703307 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-envoy\") pod \"3cbe543f-fbfb-4950-a940-8b25266ea5be\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.703363 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-token\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-token\") pod \"3cbe543f-fbfb-4950-a940-8b25266ea5be\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.703569 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-podinfo\") pod \"3cbe543f-fbfb-4950-a940-8b25266ea5be\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.703628 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cbe543f-fbfb-4950-a940-8b25266ea5be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe543f-fbfb-4950-a940-8b25266ea5be-istiod-ca-cert\") pod \"3cbe543f-fbfb-4950-a940-8b25266ea5be\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.703658 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-credential-socket\") pod \"3cbe543f-fbfb-4950-a940-8b25266ea5be\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.703691 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-data\") pod \"3cbe543f-fbfb-4950-a940-8b25266ea5be\" (UID: \"3cbe543f-fbfb-4950-a940-8b25266ea5be\") " Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.705533 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbe543f-fbfb-4950-a940-8b25266ea5be-istiod-ca-cert" pod "3cbe543f-fbfb-4950-a940-8b25266ea5be" (UID: "3cbe543f-fbfb-4950-a940-8b25266ea5be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.705671 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-certs" pod "3cbe543f-fbfb-4950-a940-8b25266ea5be" (UID: "3cbe543f-fbfb-4950-a940-8b25266ea5be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.705998 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-credential-socket" pod "3cbe543f-fbfb-4950-a940-8b25266ea5be" (UID: "3cbe543f-fbfb-4950-a940-8b25266ea5be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.709782 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-data" pod "3cbe543f-fbfb-4950-a940-8b25266ea5be" (UID: "3cbe543f-fbfb-4950-a940-8b25266ea5be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.722332 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-envoy" pod "3cbe543f-fbfb-4950-a940-8b25266ea5be" (UID: "3cbe543f-fbfb-4950-a940-8b25266ea5be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:39.735474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.735399 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-socket" pod "3cbe543f-fbfb-4950-a940-8b25266ea5be" (UID: "3cbe543f-fbfb-4950-a940-8b25266ea5be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:39.745286 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.745223 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-podinfo" pod "3cbe543f-fbfb-4950-a940-8b25266ea5be" (UID: "3cbe543f-fbfb-4950-a940-8b25266ea5be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:39.790497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.789685 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-kube-api-access-s6chn" pod "3cbe543f-fbfb-4950-a940-8b25266ea5be" (UID: "3cbe543f-fbfb-4950-a940-8b25266ea5be"). InnerVolumeSpecName "kube-api-access-s6chn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:39.796666 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.796613 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-token" pod "3cbe543f-fbfb-4950-a940-8b25266ea5be" (UID: "3cbe543f-fbfb-4950-a940-8b25266ea5be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.804741 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.804794 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.807089 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cbe543f-fbfb-4950-a940-8b25266ea5be-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.807117 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.807130 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.807143 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6chn\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-kube-api-access-s6chn\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.807155 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.807171 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.807182 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.807194 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:39.808584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.807205 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cbe543f-fbfb-4950-a940-8b25266ea5be-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:39.958578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:39.915530 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9"] Oct 30 05:55:40.235533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:40.213544 5072 cpu_manager.go:414] "containerMap: removing container" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" Oct 30 05:55:40.235533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:40.213580 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" Oct 30 05:55:40.235533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:40.214238 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" containerName="istio-proxy" Oct 30 05:55:40.849580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:40.783826 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:55:40.849580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:40.783906 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 05:55:40.953720 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:40.953671 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:41.092631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.062892 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 05:55:41.092631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.062952 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 05:55:41.160878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.160846 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.254918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.254869 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Oct 30 05:55:41.308256 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.275642 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:55:41.417581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.413542 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6fc72-d294-487e-a06b-38b7ec777cff-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.417581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.413617 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.417581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.413656 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.417581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.413733 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.417581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.413769 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.451485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.440906 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9"] Oct 30 05:55:41.452122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.452084 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" event={"ID":"3cbe543f-fbfb-4950-a940-8b25266ea5be","Type":"ContainerDied","Data":"bf7089b9afc868666baa602f7c1f1f4bc5f4b5c33489cc59003007fa4149ff78"} Oct 30 05:55:41.483009 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.482978 5072 scope.go:122] "RemoveContainer" containerID="1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55" Oct 30 05:55:41.741871 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.741777 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.742179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.742138 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.742356 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.742340 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhch\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-kube-api-access-qbhch\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.742543 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.742526 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.949725 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.981769 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhch\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-kube-api-access-qbhch\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.981868 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.982048 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6fc72-d294-487e-a06b-38b7ec777cff-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.982134 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.982189 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.982320 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.982376 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.982621 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.982677 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:41.987514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.983259 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:42.036807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.036714 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:42.038156 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.038131 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6fc72-d294-487e-a06b-38b7ec777cff-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:42.050481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.038974 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:42.171524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.087520 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 05:55:42.171524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.170552 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 05:55:42.203829 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.192236 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:42.351194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:41.949803 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 05:55:42.505111 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.505077 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8" Oct 30 05:55:42.631356 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.507602 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:42.631914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.526243 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:42.632086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.530222 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:55:42.632215 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.530270 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:55:42.657484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.632330 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 05:55:42.690307 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:42.585156 5072 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.133s" Oct 30 05:55:42.779706 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.779657 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:55:42.856332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:42.856285 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:55:43.035905 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.035793 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:43.504492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.503226 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:43.518869 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.518825 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:43.553950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.553905 5072 scope.go:122] "RemoveContainer" containerID="1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55" Oct 30 05:55:43.557785 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.557722 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhch\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-kube-api-access-qbhch\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:43.577476 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.576868 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:43.663135 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.663100 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:43.700483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.692079 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 05:55:43.700483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.692135 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 05:55:43.758598 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.745296 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:55:43.775637 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.745714 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 05:55:43.811651 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.749807 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:43.841147 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:43.812300 5072 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8_openshift-ingress_3cbe543f-fbfb-4950-a940-8b25266ea5be_0 in pod sandbox bf7089b9afc868666baa602f7c1f1f4bc5f4b5c33489cc59003007fa4149ff78 from index: no such id: '1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55'" containerID="1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55" Oct 30 05:55:43.841387 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:43.841355 5072 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8_openshift-ingress_3cbe543f-fbfb-4950-a940-8b25266ea5be_0 in pod sandbox bf7089b9afc868666baa602f7c1f1f4bc5f4b5c33489cc59003007fa4149ff78 from index: no such id: '1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55'" containerID="1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55" Oct 30 05:55:43.841641 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.840025 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:43.862714 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.862683 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 05:55:43.862993 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.862946 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 05:55:43.908615 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:43.896652 5072 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.117s" Oct 30 05:55:43.908615 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.896723 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c"] Oct 30 05:55:43.938089 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.938061 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:55:43.938423 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.938407 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c"] Oct 30 05:55:44.033571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.008244 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:55:44.033571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.008387 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:55:44.033571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:43.938360 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:55:44.054283 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.054238 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7v2r\" (UniqueName: \"kubernetes.io/projected/2f575289-1683-406b-a2c8-991d0ef9ccc4-kube-api-access-g7v2r\") pod \"payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c\" (UID: \"2f575289-1683-406b-a2c8-991d0ef9ccc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:55:44.092487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.071865 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f575289-1683-406b-a2c8-991d0ef9ccc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c\" (UID: \"2f575289-1683-406b-a2c8-991d0ef9ccc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:55:44.136937 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.136906 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:55:44.137177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.093049 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-76b905\"" Oct 30 05:55:44.137288 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.109059 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-76b905-dockercfg-2tvrt\"" Oct 30 05:55:44.168586 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.138313 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" event={"ID":"5fd3999c-39f6-4edf-9b57-71492639891d","Type":"ContainerStarted","Data":"37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6"} Oct 30 05:55:44.680605 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.240967 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f575289-1683-406b-a2c8-991d0ef9ccc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c\" (UID: \"2f575289-1683-406b-a2c8-991d0ef9ccc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:55:44.780018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.711382 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8"] Oct 30 05:55:44.780018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.711580 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-wvvc8"] Oct 30 05:55:44.780018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.713365 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:55:44.780018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.713482 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 05:55:44.874421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.874385 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 05:55:45.354604 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:45.354556 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 05:55:45.355220 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.271927 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f575289-1683-406b-a2c8-991d0ef9ccc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c\" (UID: \"2f575289-1683-406b-a2c8-991d0ef9ccc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:55:45.355477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.952609 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7v2r\" (UniqueName: \"kubernetes.io/projected/2f575289-1683-406b-a2c8-991d0ef9ccc4-kube-api-access-g7v2r\") pod \"payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c\" (UID: \"2f575289-1683-406b-a2c8-991d0ef9ccc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:55:45.355968 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:45.076648 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:45.395021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:44.912522 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:45.395583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:45.261987 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:45.649310 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:45.267050 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:55:45.669493 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:45.669274 5072 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbe543f_fbfb_4950_a940_8b25266ea5be.slice\": RecentStats: unable to find data in memory cache]" Oct 30 05:55:45.771483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:45.711564 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:55:45.771483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:45.711648 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 05:55:46.602741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:46.602692 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:46.733864 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:46.733820 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:55:46.956754 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:46.956633 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-kz9qf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.102815 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.174532 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.103840 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.112620 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.174749 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.105167 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc" podUID="3ceabc67-51ef-4f1c-86dd-88fb31892d7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.102735 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:46.961142 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-fjf2l" podUID="f1802508-d7a4-48d0-b5dc-89a480de16df" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.101740 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:55:47.205507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.175024 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 05:55:47.300085 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.300046 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cbe543f-fbfb-4950-a940-8b25266ea5be" path="/var/lib/kubelet/pods/3cbe543f-fbfb-4950-a940-8b25266ea5be/volumes" Oct 30 05:55:47.348545 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:47.331495 5072 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbe543f_fbfb_4950_a940_8b25266ea5be.slice/crio-bf7089b9afc868666baa602f7c1f1f4bc5f4b5c33489cc59003007fa4149ff78\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbe543f_fbfb_4950_a940_8b25266ea5be.slice\": RecentStats: unable to find data in memory cache]" Oct 30 05:55:47.454168 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:47.454123 5072 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.742s" Oct 30 05:55:47.591890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.581781 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jvmq9" podUID="df4f6519-1409-4d13-b154-a1313e04c7fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 30 05:55:47.591890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.585382 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:55:47.591890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.587050 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:55:47.591890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.587094 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:47.662082 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.662054 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:55:47.662790 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.662771 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:55:47.662974 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.662961 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:55:47.663145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.663125 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:55:47.663288 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.663276 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:55:47.663431 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.663420 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:55:47.663592 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.663580 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:55:47.667412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.667391 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:55:47.679145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.678038 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 05:55:47.679145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.678093 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 05:55:47.679145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.678446 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 05:55:47.679145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.678513 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 05:55:47.679145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.678679 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 05:55:47.679145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.678706 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 05:55:47.702646 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.702604 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9"] Oct 30 05:55:47.702884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.702601 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:47.703054 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.703033 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:47.703179 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.703164 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:55:47.703810 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:47.703777 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55\": container with ID starting with 1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55 not found: ID does not exist" containerID="1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55" Oct 30 05:55:47.704068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.704046 5072 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55\": container with ID starting with 1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55 not found: ID does not exist" Oct 30 05:55:47.746809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.746774 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7v2r\" (UniqueName: \"kubernetes.io/projected/2f575289-1683-406b-a2c8-991d0ef9ccc4-kube-api-access-g7v2r\") pod \"payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c\" (UID: \"2f575289-1683-406b-a2c8-991d0ef9ccc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:55:47.757002 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:47.756864 5072 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55\": container with ID starting with 1c65663354aa14bdcf43a3b978c0c8442329bcb4671f73ffa224af120dc8fe55 not found: ID does not exist" Oct 30 05:55:47.798171 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.798119 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 05:55:47.798510 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.798431 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 05:55:47.824493 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.820396 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:55:47.848369 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.843838 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:55:47.965314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:47.964345 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:55:48.427558 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.427521 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829"] Oct 30 05:55:48.428049 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.427996 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" containerID="cri-o://fa9c28a6e7c4d14a8e45a4886496746dd1417e44b8a486b078ee9b88e49f4573" gracePeriod=30 Oct 30 05:55:48.469570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.443280 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6"] Oct 30 05:55:48.470986 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.470962 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w"] Oct 30 05:55:48.471349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.471299 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" containerID="cri-o://6fcc86c96af62b9470f5159873cf2c80e36a7471d40eb60b57ce8d00dd8ffc9f" gracePeriod=30 Oct 30 05:55:48.471733 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.471689 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" containerID="cri-o://60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1" gracePeriod=30 Oct 30 05:55:48.492352 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.492295 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl"] Oct 30 05:55:48.492592 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.492548 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" containerID="cri-o://7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b" gracePeriod=30 Oct 30 05:55:48.673332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.672832 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 05:55:48.673332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.672892 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 05:55:48.865765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.861587 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" event={"ID":"bed6fc72-d294-487e-a06b-38b7ec777cff","Type":"ContainerStarted","Data":"ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388"} Oct 30 05:55:48.865765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.861676 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" event={"ID":"bed6fc72-d294-487e-a06b-38b7ec777cff","Type":"ContainerStarted","Data":"ef2d8d7050f51bee6a15b63ded990263582055ef6f0c5e0a00351d05e0ca1d4d"} Oct 30 05:55:48.888250 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.887839 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:55:48.983182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.979024 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4"] Oct 30 05:55:48.983182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.979100 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx"] Oct 30 05:55:48.983182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.979287 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" containerID="cri-o://cef277d46ad45c013742e1c289c53a449969c62cda4cc483dc843bea73f45b20" gracePeriod=30 Oct 30 05:55:48.995538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.995234 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:55:48.995538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.995406 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" containerID="cri-o://cc62c0a1387d6661e48e98e65068c998b46414481054fee719ce6a05051a94dd" gracePeriod=30 Oct 30 05:55:48.999751 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.999141 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc"] Oct 30 05:55:48.999751 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:48.999337 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" containerID="cri-o://51b56061d4a786e1e9611ba54f6b44d29c8895323564e242e27fa93a1aaddb87" gracePeriod=30 Oct 30 05:55:49.022518 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.020242 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm"] Oct 30 05:55:49.022518 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.020490 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" containerID="cri-o://d9617f2eaf4cc820d53984109f8e0f359a67f39e3829fc6caa31230c8883209a" gracePeriod=30 Oct 30 05:55:49.237225 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.236906 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5"] Oct 30 05:55:49.237801 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.237724 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" containerID="cri-o://bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b" gracePeriod=30 Oct 30 05:55:49.256822 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.256144 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc"] Oct 30 05:55:49.256822 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.256411 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" containerID="cri-o://4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a" gracePeriod=30 Oct 30 05:55:49.281322 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.281287 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn"] Oct 30 05:55:49.285569 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.284996 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" containerID="cri-o://71e21cfe4e48d19b84e0ad35614974e470ce0d978bf8894cb987bc5aa3801355" gracePeriod=30 Oct 30 05:55:49.295630 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.292891 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" podStartSLOduration=13.292874827 podStartE2EDuration="13.292874827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:49.039247932 +0000 UTC m=+1508.642937218" watchObservedRunningTime="2026-10-30 05:55:49.292874827 +0000 UTC m=+1508.896564087" Oct 30 05:55:49.296710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.296236 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42"] Oct 30 05:55:49.296710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.296511 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" containerID="cri-o://6d84558119fb5bc081798d9d481fc59cdf7ec223afe177983a954e7873a58de0" gracePeriod=30 Oct 30 05:55:49.698378 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.698334 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 05:55:49.826705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.826598 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 05:55:49.892670 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.795612 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c"] Oct 30 05:55:49.892886 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.892735 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:49.996312 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:49.970224 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:55:50.131503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:50.127451 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 05:55:50.250603 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:50.236216 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" event={"ID":"2f575289-1683-406b-a2c8-991d0ef9ccc4","Type":"ContainerStarted","Data":"f2bde00414e3e52a213fe98cf6451eaab72cec617ddab4bafde4785c2e21963d"} Oct 30 05:55:50.754491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:50.729926 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:55:50.754491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:50.731749 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:55:50.755062 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:50.754649 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:55:50.840882 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:50.840843 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:55:50.841935 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:50.841904 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 05:55:51.339608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:51.337788 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" event={"ID":"2f575289-1683-406b-a2c8-991d0ef9ccc4","Type":"ContainerStarted","Data":"41a1ad6ff1a83beff51587cbc3ba8d2538cd28430657cd2cf538a237f9c0d6ba"} Oct 30 05:55:51.339608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:51.338836 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:55:51.430536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:51.414886 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" podStartSLOduration=9.414864168 podStartE2EDuration="9.414864168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:51.395831139 +0000 UTC m=+1510.999520398" watchObservedRunningTime="2026-10-30 05:55:51.414864168 +0000 UTC m=+1511.018553430" Oct 30 05:55:51.841487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:51.841229 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:55:51.841487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:51.841289 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 05:55:52.840649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:52.840613 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:55:52.840837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:52.840680 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 05:55:53.387871 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.387838 5072 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:55:53.388330 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.387907 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:55:53.844301 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.844264 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:55:53.844673 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.844325 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 05:55:53.878500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.878452 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:55:53.967765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.967685 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw"] Oct 30 05:55:53.969266 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.969235 5072 cpu_manager.go:414] "containerMap: removing container" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" Oct 30 05:55:53.969266 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.969265 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" Oct 30 05:55:53.969565 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.969537 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="157a8419-22b9-4588-870a-0245bbefbf39" containerName="istio-proxy" Oct 30 05:55:53.973607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.973588 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:55:53.997123 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:53.997090 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw"] Oct 30 05:55:54.001582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.001557 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:55:54.014484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.009071 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-kube-api-access-qrfpj\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-kube-api-access-qrfpj\") pod \"157a8419-22b9-4588-870a-0245bbefbf39\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " Oct 30 05:55:54.014484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.009122 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-credential-socket\") pod \"157a8419-22b9-4588-870a-0245bbefbf39\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " Oct 30 05:55:54.014484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.009165 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-data\") pod \"157a8419-22b9-4588-870a-0245bbefbf39\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " Oct 30 05:55:54.014484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.009272 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-certs\") pod \"157a8419-22b9-4588-870a-0245bbefbf39\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " Oct 30 05:55:54.014484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.009327 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157a8419-22b9-4588-870a-0245bbefbf39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157a8419-22b9-4588-870a-0245bbefbf39-istiod-ca-cert\") pod \"157a8419-22b9-4588-870a-0245bbefbf39\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " Oct 30 05:55:54.014484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.009404 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-istio-token\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-istio-token\") pod \"157a8419-22b9-4588-870a-0245bbefbf39\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " Oct 30 05:55:54.014484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.009546 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-envoy\") pod \"157a8419-22b9-4588-870a-0245bbefbf39\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " Oct 30 05:55:54.014484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.009792 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-socket\") pod \"157a8419-22b9-4588-870a-0245bbefbf39\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " Oct 30 05:55:54.014484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.009833 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/157a8419-22b9-4588-870a-0245bbefbf39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157a8419-22b9-4588-870a-0245bbefbf39-istio-podinfo\") pod \"157a8419-22b9-4588-870a-0245bbefbf39\" (UID: \"157a8419-22b9-4588-870a-0245bbefbf39\") " Oct 30 05:55:54.033509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.024576 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-certs" pod "157a8419-22b9-4588-870a-0245bbefbf39" (UID: "157a8419-22b9-4588-870a-0245bbefbf39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.044515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.040738 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-credential-socket" pod "157a8419-22b9-4588-870a-0245bbefbf39" (UID: "157a8419-22b9-4588-870a-0245bbefbf39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.044515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.040945 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-data" pod "157a8419-22b9-4588-870a-0245bbefbf39" (UID: "157a8419-22b9-4588-870a-0245bbefbf39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.044515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.041368 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.044515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.041388 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.044515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.041403 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.046212 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.045214 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-socket" pod "157a8419-22b9-4588-870a-0245bbefbf39" (UID: "157a8419-22b9-4588-870a-0245bbefbf39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.046212 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.045687 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157a8419-22b9-4588-870a-0245bbefbf39-istiod-ca-cert" pod "157a8419-22b9-4588-870a-0245bbefbf39" (UID: "157a8419-22b9-4588-870a-0245bbefbf39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:54.048963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.048926 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-istio-token" pod "157a8419-22b9-4588-870a-0245bbefbf39" (UID: "157a8419-22b9-4588-870a-0245bbefbf39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:54.057974 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.057928 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/157a8419-22b9-4588-870a-0245bbefbf39-istio-podinfo" pod "157a8419-22b9-4588-870a-0245bbefbf39" (UID: "157a8419-22b9-4588-870a-0245bbefbf39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:54.058063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.058042 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-envoy" pod "157a8419-22b9-4588-870a-0245bbefbf39" (UID: "157a8419-22b9-4588-870a-0245bbefbf39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.069917 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.063619 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-kube-api-access-qrfpj" pod "157a8419-22b9-4588-870a-0245bbefbf39" (UID: "157a8419-22b9-4588-870a-0245bbefbf39"). InnerVolumeSpecName "kube-api-access-qrfpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:54.146963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.146223 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-data\") pod \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " Oct 30 05:55:54.146963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.146368 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-kube-api-access-lxf6q\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-kube-api-access-lxf6q\") pod \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " Oct 30 05:55:54.146963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.146428 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-certs\") pod \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " Oct 30 05:55:54.146963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.146644 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-credential-socket\") pod \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " Oct 30 05:55:54.146963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.146683 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-token\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-token\") pod \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " Oct 30 05:55:54.146963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.146720 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istiod-ca-cert\") pod \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " Oct 30 05:55:54.146963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.146782 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-socket\") pod \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " Oct 30 05:55:54.146963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.146813 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-envoy\") pod \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " Oct 30 05:55:54.146963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.146879 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-podinfo\") pod \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\" (UID: \"fdd3ba4a-8402-4f98-9d16-b51b61baf453\") " Oct 30 05:55:54.149473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.148641 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-data" pod "fdd3ba4a-8402-4f98-9d16-b51b61baf453" (UID: "fdd3ba4a-8402-4f98-9d16-b51b61baf453"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.149473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.148927 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-certs" pod "fdd3ba4a-8402-4f98-9d16-b51b61baf453" (UID: "fdd3ba4a-8402-4f98-9d16-b51b61baf453"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.149473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.149269 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istiod-ca-cert" pod "fdd3ba4a-8402-4f98-9d16-b51b61baf453" (UID: "fdd3ba4a-8402-4f98-9d16-b51b61baf453"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:54.149473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.149331 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-credential-socket" pod "fdd3ba4a-8402-4f98-9d16-b51b61baf453" (UID: "fdd3ba4a-8402-4f98-9d16-b51b61baf453"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.149473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.149389 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-socket" pod "fdd3ba4a-8402-4f98-9d16-b51b61baf453" (UID: "fdd3ba4a-8402-4f98-9d16-b51b61baf453"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.150450 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e382939-b971-4260-8c87-fcdad8ec96c2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw\" (UID: \"0e382939-b971-4260-8c87-fcdad8ec96c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.150854 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twq82\" (UniqueName: \"kubernetes.io/projected/0e382939-b971-4260-8c87-fcdad8ec96c2-kube-api-access-twq82\") pod \"payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw\" (UID: \"0e382939-b971-4260-8c87-fcdad8ec96c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151061 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157a8419-22b9-4588-870a-0245bbefbf39-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151077 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrfpj\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-kube-api-access-qrfpj\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151092 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151105 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157a8419-22b9-4588-870a-0245bbefbf39-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151118 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151130 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157a8419-22b9-4588-870a-0245bbefbf39-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151143 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151155 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151168 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151180 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.151307 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.151192 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157a8419-22b9-4588-870a-0245bbefbf39-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.153068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.153023 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-podinfo" pod "fdd3ba4a-8402-4f98-9d16-b51b61baf453" (UID: "fdd3ba4a-8402-4f98-9d16-b51b61baf453"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:54.155350 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.155320 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-envoy" pod "fdd3ba4a-8402-4f98-9d16-b51b61baf453" (UID: "fdd3ba4a-8402-4f98-9d16-b51b61baf453"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:54.157517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.157436 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-kube-api-access-lxf6q" pod "fdd3ba4a-8402-4f98-9d16-b51b61baf453" (UID: "fdd3ba4a-8402-4f98-9d16-b51b61baf453"). InnerVolumeSpecName "kube-api-access-lxf6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:54.165241 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.165200 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-token" pod "fdd3ba4a-8402-4f98-9d16-b51b61baf453" (UID: "fdd3ba4a-8402-4f98-9d16-b51b61baf453"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:54.184505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.183665 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb"] Oct 30 05:55:54.186703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.185113 5072 cpu_manager.go:414] "containerMap: removing container" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" Oct 30 05:55:54.186703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.185140 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" Oct 30 05:55:54.186703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.185429 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerName="istio-proxy" Oct 30 05:55:54.190086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.189504 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.190657 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.190604 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb"] Oct 30 05:55:54.270512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.270487 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b3e625-serving-cert\"" Oct 30 05:55:54.273662 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.272376 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e382939-b971-4260-8c87-fcdad8ec96c2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw\" (UID: \"0e382939-b971-4260-8c87-fcdad8ec96c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:55:54.274041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.274015 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twq82\" (UniqueName: \"kubernetes.io/projected/0e382939-b971-4260-8c87-fcdad8ec96c2-kube-api-access-twq82\") pod \"payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw\" (UID: \"0e382939-b971-4260-8c87-fcdad8ec96c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:55:54.274348 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.274332 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.274485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.274452 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.274605 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.274595 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd3ba4a-8402-4f98-9d16-b51b61baf453-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.274710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.274697 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxf6q\" (UniqueName: \"kubernetes.io/projected/fdd3ba4a-8402-4f98-9d16-b51b61baf453-kube-api-access-lxf6q\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:54.275843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.275819 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e382939-b971-4260-8c87-fcdad8ec96c2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw\" (UID: \"0e382939-b971-4260-8c87-fcdad8ec96c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:55:54.380552 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.378226 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.380552 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.378299 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af170278-15d4-44d0-9f2c-3a91ad682991-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.380552 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.378412 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sszlz\" (UniqueName: \"kubernetes.io/projected/af170278-15d4-44d0-9f2c-3a91ad682991-kube-api-access-sszlz\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.380552 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.378489 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-metrics-certs\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.385710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.385415 5072 generic.go:356] "Generic (PLEG): container finished" podID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerID="cc62c0a1387d6661e48e98e65068c998b46414481054fee719ce6a05051a94dd" exitCode=0 Oct 30 05:55:54.385710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.385539 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" event={"ID":"3ac75576-ed42-464e-b4ce-b7659ff9c7e1","Type":"ContainerDied","Data":"cc62c0a1387d6661e48e98e65068c998b46414481054fee719ce6a05051a94dd"} Oct 30 05:55:54.388703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.388539 5072 generic.go:356] "Generic (PLEG): container finished" podID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerID="fa9c28a6e7c4d14a8e45a4886496746dd1417e44b8a486b078ee9b88e49f4573" exitCode=0 Oct 30 05:55:54.388703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.388634 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" event={"ID":"cb62c6d0-9478-4a3e-b43b-9731333b9db4","Type":"ContainerDied","Data":"fa9c28a6e7c4d14a8e45a4886496746dd1417e44b8a486b078ee9b88e49f4573"} Oct 30 05:55:54.393807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.392843 5072 generic.go:356] "Generic (PLEG): container finished" podID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerID="51b56061d4a786e1e9611ba54f6b44d29c8895323564e242e27fa93a1aaddb87" exitCode=0 Oct 30 05:55:54.393807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.392979 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" event={"ID":"494ff58d-8474-487f-8bbb-a7d068c56a65","Type":"ContainerDied","Data":"51b56061d4a786e1e9611ba54f6b44d29c8895323564e242e27fa93a1aaddb87"} Oct 30 05:55:54.393807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.393033 5072 scope.go:122] "RemoveContainer" containerID="bd2f381c5ef10b2ebafd1b4bab90bc5438331606c11156831f820a2ec146073a" Oct 30 05:55:54.398063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.395770 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twq82\" (UniqueName: \"kubernetes.io/projected/0e382939-b971-4260-8c87-fcdad8ec96c2-kube-api-access-twq82\") pod \"payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw\" (UID: \"0e382939-b971-4260-8c87-fcdad8ec96c2\") " pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:55:54.400930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.400706 5072 generic.go:356] "Generic (PLEG): container finished" podID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerID="6fcc86c96af62b9470f5159873cf2c80e36a7471d40eb60b57ce8d00dd8ffc9f" exitCode=0 Oct 30 05:55:54.400930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.400836 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" event={"ID":"1e81f2fe-42c9-48c2-a934-6d45733a369f","Type":"ContainerDied","Data":"6fcc86c96af62b9470f5159873cf2c80e36a7471d40eb60b57ce8d00dd8ffc9f"} Oct 30 05:55:54.460792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.406877 5072 generic.go:356] "Generic (PLEG): container finished" podID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" containerID="7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b" exitCode=0 Oct 30 05:55:54.460792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.406984 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" event={"ID":"fdd3ba4a-8402-4f98-9d16-b51b61baf453","Type":"ContainerDied","Data":"7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b"} Oct 30 05:55:54.460792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.407037 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" event={"ID":"fdd3ba4a-8402-4f98-9d16-b51b61baf453","Type":"ContainerDied","Data":"ecf5e00006a9636e9ce14e0030df1a56d54e32066893258e191e9d4668260515"} Oct 30 05:55:54.460792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.407124 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl" Oct 30 05:55:54.491962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.481648 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-metrics-certs\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.491962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.489421 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.491962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.489536 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af170278-15d4-44d0-9f2c-3a91ad682991-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.491962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.489661 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sszlz\" (UniqueName: \"kubernetes.io/projected/af170278-15d4-44d0-9f2c-3a91ad682991-kube-api-access-sszlz\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.491962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.491335 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af170278-15d4-44d0-9f2c-3a91ad682991-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.495645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.494587 5072 generic.go:356] "Generic (PLEG): container finished" podID="157a8419-22b9-4588-870a-0245bbefbf39" containerID="60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1" exitCode=0 Oct 30 05:55:54.495645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.494754 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" event={"ID":"157a8419-22b9-4588-870a-0245bbefbf39","Type":"ContainerDied","Data":"60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1"} Oct 30 05:55:54.495645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.494804 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" event={"ID":"157a8419-22b9-4588-870a-0245bbefbf39","Type":"ContainerDied","Data":"a812f59676119497e254b54befc91fdf0312b4e52664399079cbf2a3314a6570"} Oct 30 05:55:54.495645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.494897 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:55:54.502634 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.502089 5072 generic.go:356] "Generic (PLEG): container finished" podID="43331d15-6de3-4311-995b-30953745e472" containerID="cef277d46ad45c013742e1c289c53a449969c62cda4cc483dc843bea73f45b20" exitCode=0 Oct 30 05:55:54.502634 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.502185 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" event={"ID":"43331d15-6de3-4311-995b-30953745e472","Type":"ContainerDied","Data":"cef277d46ad45c013742e1c289c53a449969c62cda4cc483dc843bea73f45b20"} Oct 30 05:55:54.505359 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.505161 5072 generic.go:356] "Generic (PLEG): container finished" podID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerID="d9617f2eaf4cc820d53984109f8e0f359a67f39e3829fc6caa31230c8883209a" exitCode=0 Oct 30 05:55:54.505359 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.505291 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" event={"ID":"f05e3600-3aa0-4c56-97ec-5fa829f18c7e","Type":"ContainerDied","Data":"d9617f2eaf4cc820d53984109f8e0f359a67f39e3829fc6caa31230c8883209a"} Oct 30 05:55:54.566954 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.563399 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sszlz\" (UniqueName: \"kubernetes.io/projected/af170278-15d4-44d0-9f2c-3a91ad682991-kube-api-access-sszlz\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.566954 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.564860 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-metrics-certs\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.566954 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.566215 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-maas-api-tls\") pod \"maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.601492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.601450 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:55:54.662974 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.626556 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:55:54.662974 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.626618 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 05:55:54.806788 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.806763 5072 scope.go:122] "RemoveContainer" containerID="7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b" Oct 30 05:55:54.859191 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.854128 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:55:54.859191 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.854201 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 05:55:54.864813 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.864629 5072 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:55:54.864813 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.864681 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 05:55:54.885565 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.885538 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:55:54.885816 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.885798 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:54.889693 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.889670 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:55:54.905050 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.904620 5072 scope.go:122] "RemoveContainer" containerID="7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b" Oct 30 05:55:55.104589 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:54.998641 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b\": container with ID starting with 7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b not found: ID does not exist" containerID="7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b" Oct 30 05:55:55.104589 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.998700 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b"} err="failed to get container status \"7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b\": rpc error: code = NotFound desc = could not find container \"7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b\": container with ID starting with 7da3fb3d534fab7017cf28abf1247c9af914cbfa08299a81650ca34c73cdab6b not found: ID does not exist" Oct 30 05:55:55.104589 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:54.998758 5072 scope.go:122] "RemoveContainer" containerID="60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1" Oct 30 05:55:55.177080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177031 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-data\") pod \"1e81f2fe-42c9-48c2-a934-6d45733a369f\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " Oct 30 05:55:55.177271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177252 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-token\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-token\") pod \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " Oct 30 05:55:55.177362 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177345 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-credential-socket\") pod \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " Oct 30 05:55:55.177420 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177388 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-podinfo\") pod \"1e81f2fe-42c9-48c2-a934-6d45733a369f\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " Oct 30 05:55:55.177495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177439 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-envoy\") pod \"1e81f2fe-42c9-48c2-a934-6d45733a369f\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " Oct 30 05:55:55.177613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177509 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-podinfo\") pod \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " Oct 30 05:55:55.177613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177555 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-credential-socket\") pod \"1e81f2fe-42c9-48c2-a934-6d45733a369f\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " Oct 30 05:55:55.177613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177598 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-certs\") pod \"1e81f2fe-42c9-48c2-a934-6d45733a369f\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " Oct 30 05:55:55.177768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177639 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-kube-api-access-8lpvv\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-kube-api-access-8lpvv\") pod \"1e81f2fe-42c9-48c2-a934-6d45733a369f\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " Oct 30 05:55:55.177768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177673 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-token\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-token\") pod \"1e81f2fe-42c9-48c2-a934-6d45733a369f\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " Oct 30 05:55:55.177768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177733 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-socket\") pod \"1e81f2fe-42c9-48c2-a934-6d45733a369f\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177910 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-certs\") pod \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.177959 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-socket\") pod \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.178013 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-data\") pod \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.178059 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istiod-ca-cert\") pod \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.178156 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-envoy\") pod \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.178214 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e81f2fe-42c9-48c2-a934-6d45733a369f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81f2fe-42c9-48c2-a934-6d45733a369f-istiod-ca-cert\") pod \"1e81f2fe-42c9-48c2-a934-6d45733a369f\" (UID: \"1e81f2fe-42c9-48c2-a934-6d45733a369f\") " Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.178369 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-kube-api-access-gfwvc\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-kube-api-access-gfwvc\") pod \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\" (UID: \"cb62c6d0-9478-4a3e-b43b-9731333b9db4\") " Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.188712 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-socket" pod "1e81f2fe-42c9-48c2-a934-6d45733a369f" (UID: "1e81f2fe-42c9-48c2-a934-6d45733a369f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.188914 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-data" pod "1e81f2fe-42c9-48c2-a934-6d45733a369f" (UID: "1e81f2fe-42c9-48c2-a934-6d45733a369f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.194122 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-socket" pod "cb62c6d0-9478-4a3e-b43b-9731333b9db4" (UID: "cb62c6d0-9478-4a3e-b43b-9731333b9db4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.194373 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-data" pod "cb62c6d0-9478-4a3e-b43b-9731333b9db4" (UID: "cb62c6d0-9478-4a3e-b43b-9731333b9db4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.194704 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istiod-ca-cert" pod "cb62c6d0-9478-4a3e-b43b-9731333b9db4" (UID: "cb62c6d0-9478-4a3e-b43b-9731333b9db4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.195816 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-credential-socket" pod "cb62c6d0-9478-4a3e-b43b-9731333b9db4" (UID: "cb62c6d0-9478-4a3e-b43b-9731333b9db4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.195999 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-certs" pod "cb62c6d0-9478-4a3e-b43b-9731333b9db4" (UID: "cb62c6d0-9478-4a3e-b43b-9731333b9db4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.196595 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e81f2fe-42c9-48c2-a934-6d45733a369f-istiod-ca-cert" pod "1e81f2fe-42c9-48c2-a934-6d45733a369f" (UID: "1e81f2fe-42c9-48c2-a934-6d45733a369f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.202584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.197282 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-credential-socket" pod "1e81f2fe-42c9-48c2-a934-6d45733a369f" (UID: "1e81f2fe-42c9-48c2-a934-6d45733a369f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.218696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.215109 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-podinfo" pod "1e81f2fe-42c9-48c2-a934-6d45733a369f" (UID: "1e81f2fe-42c9-48c2-a934-6d45733a369f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.218696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.215230 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-token" pod "cb62c6d0-9478-4a3e-b43b-9731333b9db4" (UID: "cb62c6d0-9478-4a3e-b43b-9731333b9db4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.218696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.215356 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-envoy" pod "1e81f2fe-42c9-48c2-a934-6d45733a369f" (UID: "1e81f2fe-42c9-48c2-a934-6d45733a369f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.241034 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.241006 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-certs" pod "1e81f2fe-42c9-48c2-a934-6d45733a369f" (UID: "1e81f2fe-42c9-48c2-a934-6d45733a369f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.251278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.251248 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl"] Oct 30 05:55:55.254998 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.254954 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-kube-api-access-8lpvv" pod "1e81f2fe-42c9-48c2-a934-6d45733a369f" (UID: "1e81f2fe-42c9-48c2-a934-6d45733a369f"). InnerVolumeSpecName "kube-api-access-8lpvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.255561 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.255524 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-kube-api-access-gfwvc" pod "cb62c6d0-9478-4a3e-b43b-9731333b9db4" (UID: "cb62c6d0-9478-4a3e-b43b-9731333b9db4"). InnerVolumeSpecName "kube-api-access-gfwvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.256387 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.256338 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-envoy" pod "cb62c6d0-9478-4a3e-b43b-9731333b9db4" (UID: "cb62c6d0-9478-4a3e-b43b-9731333b9db4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.257918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.257244 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-podinfo" pod "cb62c6d0-9478-4a3e-b43b-9731333b9db4" (UID: "cb62c6d0-9478-4a3e-b43b-9731333b9db4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.263726 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.263691 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-token" pod "1e81f2fe-42c9-48c2-a934-6d45733a369f" (UID: "1e81f2fe-42c9-48c2-a934-6d45733a369f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.268010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.266529 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-d9qdl"] Oct 30 05:55:55.278951 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.278742 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:55:55.291558 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.291425 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.297946 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfwvc\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-kube-api-access-gfwvc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.297968 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.297979 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.297989 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.297996 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298004 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298011 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298019 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298028 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298037 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpvv\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-kube-api-access-8lpvv\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298044 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e81f2fe-42c9-48c2-a934-6d45733a369f-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298052 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81f2fe-42c9-48c2-a934-6d45733a369f-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298060 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298068 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298076 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298083 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298090 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb62c6d0-9478-4a3e-b43b-9731333b9db4-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.298114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.298097 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81f2fe-42c9-48c2-a934-6d45733a369f-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.309274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.309178 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:55:55.322886 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.322768 5072 scope.go:122] "RemoveContainer" containerID="60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1" Oct 30 05:55:55.324434 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:55.324236 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1\": container with ID starting with 60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1 not found: ID does not exist" containerID="60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1" Oct 30 05:55:55.324434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.324276 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1"} err="failed to get container status \"60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1\": rpc error: code = NotFound desc = could not find container \"60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1\": container with ID starting with 60c83f696c60a907a58949078c9709f1e50348f8cc1962f431847bff6c3422d1 not found: ID does not exist" Oct 30 05:55:55.398928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.398877 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-socket\") pod \"494ff58d-8474-487f-8bbb-a7d068c56a65\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.398931 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-kube-api-access-pqbdp\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-kube-api-access-pqbdp\") pod \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.398967 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-podinfo\") pod \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399005 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-token\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-token\") pod \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399056 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-credential-socket\") pod \"494ff58d-8474-487f-8bbb-a7d068c56a65\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399091 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-data\") pod \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399136 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-kube-api-access-88ghr\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-kube-api-access-88ghr\") pod \"494ff58d-8474-487f-8bbb-a7d068c56a65\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399173 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-envoy\") pod \"43331d15-6de3-4311-995b-30953745e472\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399219 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43331d15-6de3-4311-995b-30953745e472-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43331d15-6de3-4311-995b-30953745e472-istiod-ca-cert\") pod \"43331d15-6de3-4311-995b-30953745e472\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399294 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-envoy\") pod \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " Oct 30 05:55:55.399376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399375 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-data\") pod \"494ff58d-8474-487f-8bbb-a7d068c56a65\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399451 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-istio-token\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-istio-token\") pod \"43331d15-6de3-4311-995b-30953745e472\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399498 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istiod-ca-cert\") pod \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399622 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-envoy\") pod \"494ff58d-8474-487f-8bbb-a7d068c56a65\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399649 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-certs\") pod \"43331d15-6de3-4311-995b-30953745e472\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399686 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-data\") pod \"43331d15-6de3-4311-995b-30953745e472\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399775 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-credential-socket\") pod \"43331d15-6de3-4311-995b-30953745e472\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399817 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-socket\") pod \"43331d15-6de3-4311-995b-30953745e472\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399850 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-certs\") pod \"494ff58d-8474-487f-8bbb-a7d068c56a65\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399898 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-certs\") pod \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.399943 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-podinfo\") pod \"494ff58d-8474-487f-8bbb-a7d068c56a65\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.400035 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-credential-socket\") pod \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.400098 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-socket\") pod \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\" (UID: \"f05e3600-3aa0-4c56-97ec-5fa829f18c7e\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.400147 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-token\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-token\") pod \"494ff58d-8474-487f-8bbb-a7d068c56a65\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.400212 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494ff58d-8474-487f-8bbb-a7d068c56a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ff58d-8474-487f-8bbb-a7d068c56a65-istiod-ca-cert\") pod \"494ff58d-8474-487f-8bbb-a7d068c56a65\" (UID: \"494ff58d-8474-487f-8bbb-a7d068c56a65\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.400244 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43331d15-6de3-4311-995b-30953745e472-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43331d15-6de3-4311-995b-30953745e472-istio-podinfo\") pod \"43331d15-6de3-4311-995b-30953745e472\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.400278 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-kube-api-access-48dxc\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-kube-api-access-48dxc\") pod \"43331d15-6de3-4311-995b-30953745e472\" (UID: \"43331d15-6de3-4311-995b-30953745e472\") " Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.402363 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43331d15-6de3-4311-995b-30953745e472-istiod-ca-cert" pod "43331d15-6de3-4311-995b-30953745e472" (UID: "43331d15-6de3-4311-995b-30953745e472"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.403291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.402905 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-socket" pod "494ff58d-8474-487f-8bbb-a7d068c56a65" (UID: "494ff58d-8474-487f-8bbb-a7d068c56a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.405217 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.404880 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-socket" pod "43331d15-6de3-4311-995b-30953745e472" (UID: "43331d15-6de3-4311-995b-30953745e472"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.406017 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.405801 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-credential-socket" pod "f05e3600-3aa0-4c56-97ec-5fa829f18c7e" (UID: "f05e3600-3aa0-4c56-97ec-5fa829f18c7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.406325 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.406164 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-certs" pod "494ff58d-8474-487f-8bbb-a7d068c56a65" (UID: "494ff58d-8474-487f-8bbb-a7d068c56a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.406715 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.406445 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-certs" pod "f05e3600-3aa0-4c56-97ec-5fa829f18c7e" (UID: "f05e3600-3aa0-4c56-97ec-5fa829f18c7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.409772 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.409756 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43331d15-6de3-4311-995b-30953745e472-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.409869 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.409857 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.409950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.409938 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.410029 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.410019 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.410114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.410103 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.410256 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.410243 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.416048 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.415545 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-certs" pod "43331d15-6de3-4311-995b-30953745e472" (UID: "43331d15-6de3-4311-995b-30953745e472"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.416048 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.415749 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-data" pod "43331d15-6de3-4311-995b-30953745e472" (UID: "43331d15-6de3-4311-995b-30953745e472"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.416790 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.416514 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-socket" pod "f05e3600-3aa0-4c56-97ec-5fa829f18c7e" (UID: "f05e3600-3aa0-4c56-97ec-5fa829f18c7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.420020 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.419904 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istiod-ca-cert" pod "f05e3600-3aa0-4c56-97ec-5fa829f18c7e" (UID: "f05e3600-3aa0-4c56-97ec-5fa829f18c7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.420020 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.419954 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-data" pod "f05e3600-3aa0-4c56-97ec-5fa829f18c7e" (UID: "f05e3600-3aa0-4c56-97ec-5fa829f18c7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.420020 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.419994 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-credential-socket" pod "43331d15-6de3-4311-995b-30953745e472" (UID: "43331d15-6de3-4311-995b-30953745e472"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.425167 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.424590 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-data" pod "494ff58d-8474-487f-8bbb-a7d068c56a65" (UID: "494ff58d-8474-487f-8bbb-a7d068c56a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.425167 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.424913 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494ff58d-8474-487f-8bbb-a7d068c56a65-istiod-ca-cert" pod "494ff58d-8474-487f-8bbb-a7d068c56a65" (UID: "494ff58d-8474-487f-8bbb-a7d068c56a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.432765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.432651 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-envoy" pod "43331d15-6de3-4311-995b-30953745e472" (UID: "43331d15-6de3-4311-995b-30953745e472"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.432765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.432723 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43331d15-6de3-4311-995b-30953745e472-istio-podinfo" pod "43331d15-6de3-4311-995b-30953745e472" (UID: "43331d15-6de3-4311-995b-30953745e472"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.450478 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.450348 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-credential-socket" pod "494ff58d-8474-487f-8bbb-a7d068c56a65" (UID: "494ff58d-8474-487f-8bbb-a7d068c56a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.469551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.468230 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:55:55.469551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.469499 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-token" pod "f05e3600-3aa0-4c56-97ec-5fa829f18c7e" (UID: "f05e3600-3aa0-4c56-97ec-5fa829f18c7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.470845 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.470777 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-envoy" pod "494ff58d-8474-487f-8bbb-a7d068c56a65" (UID: "494ff58d-8474-487f-8bbb-a7d068c56a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.470845 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.470818 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-istio-token" pod "43331d15-6de3-4311-995b-30953745e472" (UID: "43331d15-6de3-4311-995b-30953745e472"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.471296 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.470907 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-kube-api-access-48dxc" pod "43331d15-6de3-4311-995b-30953745e472" (UID: "43331d15-6de3-4311-995b-30953745e472"). InnerVolumeSpecName "kube-api-access-48dxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.471597 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.471485 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-kube-api-access-pqbdp" pod "f05e3600-3aa0-4c56-97ec-5fa829f18c7e" (UID: "f05e3600-3aa0-4c56-97ec-5fa829f18c7e"). InnerVolumeSpecName "kube-api-access-pqbdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.471597 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.471562 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-envoy" pod "f05e3600-3aa0-4c56-97ec-5fa829f18c7e" (UID: "f05e3600-3aa0-4c56-97ec-5fa829f18c7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.472340 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.472313 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-podinfo" pod "494ff58d-8474-487f-8bbb-a7d068c56a65" (UID: "494ff58d-8474-487f-8bbb-a7d068c56a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.473918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.473884 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-token" pod "494ff58d-8474-487f-8bbb-a7d068c56a65" (UID: "494ff58d-8474-487f-8bbb-a7d068c56a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.474701 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.474662 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-kube-api-access-88ghr" pod "494ff58d-8474-487f-8bbb-a7d068c56a65" (UID: "494ff58d-8474-487f-8bbb-a7d068c56a65"). InnerVolumeSpecName "kube-api-access-88ghr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.475354 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.475175 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-podinfo" pod "f05e3600-3aa0-4c56-97ec-5fa829f18c7e" (UID: "f05e3600-3aa0-4c56-97ec-5fa829f18c7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.531608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.531572 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-token\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-token\") pod \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " Oct 30 05:55:55.531975 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.531954 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-envoy\") pod \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " Oct 30 05:55:55.540356 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.540329 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:55:55.540653 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.540629 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-podinfo\") pod \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " Oct 30 05:55:55.541207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.540793 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-socket\") pod \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " Oct 30 05:55:55.541207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.540851 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istiod-ca-cert\") pod \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " Oct 30 05:55:55.541207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.540951 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-data\") pod \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " Oct 30 05:55:55.541207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.541047 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-kube-api-access-r9wcw\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-kube-api-access-r9wcw\") pod \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " Oct 30 05:55:55.541207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.541094 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-certs\") pod \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " Oct 30 05:55:55.541207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.541129 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-credential-socket\") pod \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\" (UID: \"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad\") " Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.541979 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-data" pod "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" (UID: "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542179 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-credential-socket" pod "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" (UID: "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542388 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-socket" pod "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" (UID: "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542412 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542430 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542444 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542482 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542494 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542507 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ff58d-8474-487f-8bbb-a7d068c56a65-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542518 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43331d15-6de3-4311-995b-30953745e472-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542534 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48dxc\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-kube-api-access-48dxc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542548 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqbdp\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-kube-api-access-pqbdp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542560 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542573 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542589 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542601 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542612 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542624 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88ghr\" (UniqueName: \"kubernetes.io/projected/494ff58d-8474-487f-8bbb-a7d068c56a65-kube-api-access-88ghr\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542640 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542652 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542664 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542676 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542714 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43331d15-6de3-4311-995b-30953745e472-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542726 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05e3600-3aa0-4c56-97ec-5fa829f18c7e-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542739 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ff58d-8474-487f-8bbb-a7d068c56a65-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.542815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.542752 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43331d15-6de3-4311-995b-30953745e472-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.547641 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.544963 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-certs" pod "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" (UID: "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.547641 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.547509 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istiod-ca-cert" pod "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" (UID: "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.562663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.554275 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-envoy" pod "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" (UID: "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.562663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.554971 5072 generic.go:356] "Generic (PLEG): container finished" podID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerID="4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a" exitCode=0 Oct 30 05:55:55.562663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.555073 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" event={"ID":"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad","Type":"ContainerDied","Data":"4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a"} Oct 30 05:55:55.562663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.555137 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" event={"ID":"8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad","Type":"ContainerDied","Data":"b7cd074e64ed436f486f7cf57936146770a455889fe1afed97568503e9e695a3"} Oct 30 05:55:55.562663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.555176 5072 scope.go:122] "RemoveContainer" containerID="4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a" Oct 30 05:55:55.562663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.555373 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc" Oct 30 05:55:55.562663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.555820 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-podinfo" pod "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" (UID: "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.562663 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.560500 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-kube-api-access-r9wcw" pod "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" (UID: "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad"). InnerVolumeSpecName "kube-api-access-r9wcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.593953 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.589437 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-token" pod "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" (UID: "8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.621575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.618957 5072 generic.go:356] "Generic (PLEG): container finished" podID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerID="71e21cfe4e48d19b84e0ad35614974e470ce0d978bf8894cb987bc5aa3801355" exitCode=0 Oct 30 05:55:55.621575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.619173 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" event={"ID":"117dc795-66d4-478c-9ff7-b23b056a4d48","Type":"ContainerDied","Data":"71e21cfe4e48d19b84e0ad35614974e470ce0d978bf8894cb987bc5aa3801355"} Oct 30 05:55:55.628022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.627240 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" event={"ID":"3ac75576-ed42-464e-b4ce-b7659ff9c7e1","Type":"ContainerDied","Data":"c8bd1a1c2bb058a777e712a62188606b85bd3e5c0b5ff46b0ec33383968de767"} Oct 30 05:55:55.628022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.627346 5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8bd1a1c2bb058a777e712a62188606b85bd3e5c0b5ff46b0ec33383968de767" Oct 30 05:55:55.631513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.629369 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" event={"ID":"cb62c6d0-9478-4a3e-b43b-9731333b9db4","Type":"ContainerDied","Data":"213e0726cfad2f8bffc012a1d0bfcc126d34a241637a0346ed830845cff730ca"} Oct 30 05:55:55.631513 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.629544 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829" Oct 30 05:55:55.633770 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.633677 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" event={"ID":"494ff58d-8474-487f-8bbb-a7d068c56a65","Type":"ContainerDied","Data":"79a09ad4022290a7568c458d1aa8b012b5310863048206469191cef34fce7f57"} Oct 30 05:55:55.634166 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.634147 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc" Oct 30 05:55:55.639937 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.639911 5072 generic.go:356] "Generic (PLEG): container finished" podID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerID="6d84558119fb5bc081798d9d481fc59cdf7ec223afe177983a954e7873a58de0" exitCode=0 Oct 30 05:55:55.640022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.640003 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" event={"ID":"82915e36-0e03-48c5-8d83-942037d0e5f5","Type":"ContainerDied","Data":"6d84558119fb5bc081798d9d481fc59cdf7ec223afe177983a954e7873a58de0"} Oct 30 05:55:55.641696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.641560 5072 generic.go:356] "Generic (PLEG): container finished" podID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerID="bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b" exitCode=0 Oct 30 05:55:55.642108 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.641839 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" event={"ID":"e7caf44c-c7f8-4daa-8957-1ea7b2729676","Type":"ContainerDied","Data":"bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b"} Oct 30 05:55:55.642108 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.641888 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" event={"ID":"e7caf44c-c7f8-4daa-8957-1ea7b2729676","Type":"ContainerDied","Data":"900ce09f4e4ea3e9b2e60c3fdbb67233af3d91189adfb4724025ad7df110cdb1"} Oct 30 05:55:55.642108 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.641980 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5" Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.643334 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-envoy\") pod \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.643553 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-podinfo\") pod \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.643644 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-token\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-token\") pod \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.643687 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-data\") pod \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.643742 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-socket\") pod \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.643820 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-kube-api-access-wh2bp\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-kube-api-access-wh2bp\") pod \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.643869 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-credential-socket\") pod \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.643926 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-certs\") pod \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.644026 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istiod-ca-cert\") pod \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\" (UID: \"e7caf44c-c7f8-4daa-8957-1ea7b2729676\") " Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.645000 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9wcw\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-kube-api-access-r9wcw\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.645021 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.645036 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.645049 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.645062 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.645109 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.645812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.645128 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.649032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.647278 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-credential-socket" pod "e7caf44c-c7f8-4daa-8957-1ea7b2729676" (UID: "e7caf44c-c7f8-4daa-8957-1ea7b2729676"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.649032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.647565 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istiod-ca-cert" pod "e7caf44c-c7f8-4daa-8957-1ea7b2729676" (UID: "e7caf44c-c7f8-4daa-8957-1ea7b2729676"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.649032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.647835 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-data" pod "e7caf44c-c7f8-4daa-8957-1ea7b2729676" (UID: "e7caf44c-c7f8-4daa-8957-1ea7b2729676"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.649032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.648038 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-certs" pod "e7caf44c-c7f8-4daa-8957-1ea7b2729676" (UID: "e7caf44c-c7f8-4daa-8957-1ea7b2729676"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.649032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.648868 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" event={"ID":"1e81f2fe-42c9-48c2-a934-6d45733a369f","Type":"ContainerDied","Data":"46ce22ff87c72ca1a49c22fc961c98e22dd4e349d8454e99dd0ee9722727529a"} Oct 30 05:55:55.649032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.648977 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w" Oct 30 05:55:55.649429 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.649141 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-socket" pod "e7caf44c-c7f8-4daa-8957-1ea7b2729676" (UID: "e7caf44c-c7f8-4daa-8957-1ea7b2729676"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.666959 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.666922 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" event={"ID":"43331d15-6de3-4311-995b-30953745e472","Type":"ContainerDied","Data":"5b812d98dc8cb2fca39ac85361e0bd998924a5762cce38eec0e9f5b820b948a2"} Oct 30 05:55:55.667070 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.667056 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx" Oct 30 05:55:55.669222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.669181 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-podinfo" pod "e7caf44c-c7f8-4daa-8957-1ea7b2729676" (UID: "e7caf44c-c7f8-4daa-8957-1ea7b2729676"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.670608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.670507 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-kube-api-access-wh2bp" pod "e7caf44c-c7f8-4daa-8957-1ea7b2729676" (UID: "e7caf44c-c7f8-4daa-8957-1ea7b2729676"). InnerVolumeSpecName "kube-api-access-wh2bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.671594 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.671570 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-envoy" pod "e7caf44c-c7f8-4daa-8957-1ea7b2729676" (UID: "e7caf44c-c7f8-4daa-8957-1ea7b2729676"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.673051 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.673017 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-token" pod "e7caf44c-c7f8-4daa-8957-1ea7b2729676" (UID: "e7caf44c-c7f8-4daa-8957-1ea7b2729676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.683073 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.673395 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" event={"ID":"f05e3600-3aa0-4c56-97ec-5fa829f18c7e","Type":"ContainerDied","Data":"cf43aaf58719976c2420f296af3a2703ee6ff347d51dc5fe60159569b25d2665"} Oct 30 05:55:55.683073 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.673451 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm" Oct 30 05:55:55.713741 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.713717 5072 scope.go:122] "RemoveContainer" containerID="4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a" Oct 30 05:55:55.714158 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:55.714135 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a\": container with ID starting with 4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a not found: ID does not exist" containerID="4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a" Oct 30 05:55:55.714313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.714167 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a"} err="failed to get container status \"4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a\": rpc error: code = NotFound desc = could not find container \"4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a\": container with ID starting with 4b93fdcc1d7808c4dd4e00e3dba221669d25a4525fe8ae0b4f693632c695f68a not found: ID does not exist" Oct 30 05:55:55.714313 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.714185 5072 scope.go:122] "RemoveContainer" containerID="fa9c28a6e7c4d14a8e45a4886496746dd1417e44b8a486b078ee9b88e49f4573" Oct 30 05:55:55.722568 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.722550 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:55:55.736555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.736534 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:55:55.737168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.737151 5072 scope.go:122] "RemoveContainer" containerID="51b56061d4a786e1e9611ba54f6b44d29c8895323564e242e27fa93a1aaddb87" Oct 30 05:55:55.741967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.741945 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:55:55.745645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.745620 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-data\") pod \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " Oct 30 05:55:55.745782 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.745769 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-envoy\") pod \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " Oct 30 05:55:55.745858 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.745846 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-kube-api-access-g5xtn\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-kube-api-access-g5xtn\") pod \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " Oct 30 05:55:55.745944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.745875 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-socket\") pod \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " Oct 30 05:55:55.745997 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.745984 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istiod-ca-cert\") pod \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " Oct 30 05:55:55.746043 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.746025 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-token\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-token\") pod \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " Oct 30 05:55:55.746158 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.746143 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-podinfo\") pod \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " Oct 30 05:55:55.746226 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.746190 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-certs\") pod \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " Oct 30 05:55:55.746297 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.746231 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-credential-socket\") pod \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\" (UID: \"3ac75576-ed42-464e-b4ce-b7659ff9c7e1\") " Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.746857 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-data" pod "3ac75576-ed42-464e-b4ce-b7659ff9c7e1" (UID: "3ac75576-ed42-464e-b4ce-b7659ff9c7e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747193 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747211 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747219 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747228 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747237 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747249 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747258 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2bp\" (UniqueName: \"kubernetes.io/projected/e7caf44c-c7f8-4daa-8957-1ea7b2729676-kube-api-access-wh2bp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747267 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747275 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7caf44c-c7f8-4daa-8957-1ea7b2729676-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.747587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747283 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7caf44c-c7f8-4daa-8957-1ea7b2729676-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.748154 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747669 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-socket" pod "3ac75576-ed42-464e-b4ce-b7659ff9c7e1" (UID: "3ac75576-ed42-464e-b4ce-b7659ff9c7e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.748154 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.747873 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istiod-ca-cert" pod "3ac75576-ed42-464e-b4ce-b7659ff9c7e1" (UID: "3ac75576-ed42-464e-b4ce-b7659ff9c7e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.748267 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.748214 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-credential-socket" pod "3ac75576-ed42-464e-b4ce-b7659ff9c7e1" (UID: "3ac75576-ed42-464e-b4ce-b7659ff9c7e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.748317 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.748266 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-certs" pod "3ac75576-ed42-464e-b4ce-b7659ff9c7e1" (UID: "3ac75576-ed42-464e-b4ce-b7659ff9c7e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.752635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.752613 5072 scope.go:122] "RemoveContainer" containerID="bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b" Oct 30 05:55:55.753842 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.753819 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-podinfo" pod "3ac75576-ed42-464e-b4ce-b7659ff9c7e1" (UID: "3ac75576-ed42-464e-b4ce-b7659ff9c7e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.753934 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.753917 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-envoy" pod "3ac75576-ed42-464e-b4ce-b7659ff9c7e1" (UID: "3ac75576-ed42-464e-b4ce-b7659ff9c7e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.754057 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.754041 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-kube-api-access-g5xtn" pod "3ac75576-ed42-464e-b4ce-b7659ff9c7e1" (UID: "3ac75576-ed42-464e-b4ce-b7659ff9c7e1"). InnerVolumeSpecName "kube-api-access-g5xtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.755987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.755966 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-token" pod "3ac75576-ed42-464e-b4ce-b7659ff9c7e1" (UID: "3ac75576-ed42-464e-b4ce-b7659ff9c7e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.774416 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.774395 5072 scope.go:122] "RemoveContainer" containerID="bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b" Oct 30 05:55:55.774826 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:55:55.774796 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b\": container with ID starting with bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b not found: ID does not exist" containerID="bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b" Oct 30 05:55:55.774926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.774833 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b"} err="failed to get container status \"bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b\": rpc error: code = NotFound desc = could not find container \"bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b\": container with ID starting with bc214fb18c8e9312f70853814b5cc0cfc4d63d81b96ba59b0b8c3e19c663842b not found: ID does not exist" Oct 30 05:55:55.774926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.774857 5072 scope.go:122] "RemoveContainer" containerID="6fcc86c96af62b9470f5159873cf2c80e36a7471d40eb60b57ce8d00dd8ffc9f" Oct 30 05:55:55.789219 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.789194 5072 scope.go:122] "RemoveContainer" containerID="cef277d46ad45c013742e1c289c53a449969c62cda4cc483dc843bea73f45b20" Oct 30 05:55:55.803645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.803619 5072 scope.go:122] "RemoveContainer" containerID="d9617f2eaf4cc820d53984109f8e0f359a67f39e3829fc6caa31230c8883209a" Oct 30 05:55:55.825654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.825615 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw"] Oct 30 05:55:55.830011 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:55:55.829970 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e382939_b971_4260_8c87_fcdad8ec96c2.slice/crio-d6af83c08c2aaf7c4aa33752fefe41b4dd0c2c035853fd93935260a5f77be5f4 WatchSource:0}: Error finding container d6af83c08c2aaf7c4aa33752fefe41b4dd0c2c035853fd93935260a5f77be5f4: Status 404 returned error can't find the container with id d6af83c08c2aaf7c4aa33752fefe41b4dd0c2c035853fd93935260a5f77be5f4 Oct 30 05:55:55.830341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.830320 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm"] Oct 30 05:55:55.837158 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.837132 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-v4tbm"] Oct 30 05:55:55.843691 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.843667 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb"] Oct 30 05:55:55.845941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.845921 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:55.846325 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.846295 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:55.847098 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.847079 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:55:55.847416 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:55:55.847389 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf170278_15d4_44d0_9f2c_3a91ad682991.slice/crio-96f9db896d69b8142c1049982004e195a0e5186fc2538994c22b18bbdabf07a9 WatchSource:0}: Error finding container 96f9db896d69b8142c1049982004e195a0e5186fc2538994c22b18bbdabf07a9: Status 404 returned error can't find the container with id 96f9db896d69b8142c1049982004e195a0e5186fc2538994c22b18bbdabf07a9 Oct 30 05:55:55.847802 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.847783 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-socket\") pod \"117dc795-66d4-478c-9ff7-b23b056a4d48\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " Oct 30 05:55:55.847884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.847853 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-envoy\") pod \"82915e36-0e03-48c5-8d83-942037d0e5f5\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " Oct 30 05:55:55.847930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.847901 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-socket\") pod \"82915e36-0e03-48c5-8d83-942037d0e5f5\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " Oct 30 05:55:55.847970 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.847955 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-envoy\") pod \"117dc795-66d4-478c-9ff7-b23b056a4d48\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " Oct 30 05:55:55.848028 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.847994 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-data\") pod \"82915e36-0e03-48c5-8d83-942037d0e5f5\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " Oct 30 05:55:55.848081 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848032 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-token\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-token\") pod \"82915e36-0e03-48c5-8d83-942037d0e5f5\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " Oct 30 05:55:55.848081 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848059 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-socket" pod "117dc795-66d4-478c-9ff7-b23b056a4d48" (UID: "117dc795-66d4-478c-9ff7-b23b056a4d48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.848196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848095 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82915e36-0e03-48c5-8d83-942037d0e5f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82915e36-0e03-48c5-8d83-942037d0e5f5-istiod-ca-cert\") pod \"82915e36-0e03-48c5-8d83-942037d0e5f5\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " Oct 30 05:55:55.848196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848136 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-kube-api-access-q2fhl\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-kube-api-access-q2fhl\") pod \"82915e36-0e03-48c5-8d83-942037d0e5f5\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " Oct 30 05:55:55.848196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848185 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-certs\") pod \"82915e36-0e03-48c5-8d83-942037d0e5f5\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " Oct 30 05:55:55.848412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848252 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-socket" pod "82915e36-0e03-48c5-8d83-942037d0e5f5" (UID: "82915e36-0e03-48c5-8d83-942037d0e5f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.848412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848315 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-credential-socket\") pod \"117dc795-66d4-478c-9ff7-b23b056a4d48\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " Oct 30 05:55:55.848412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848320 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-data" pod "82915e36-0e03-48c5-8d83-942037d0e5f5" (UID: "82915e36-0e03-48c5-8d83-942037d0e5f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.848412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848346 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117dc795-66d4-478c-9ff7-b23b056a4d48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117dc795-66d4-478c-9ff7-b23b056a4d48-istiod-ca-cert\") pod \"117dc795-66d4-478c-9ff7-b23b056a4d48\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " Oct 30 05:55:55.848674 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848410 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-certs\") pod \"117dc795-66d4-478c-9ff7-b23b056a4d48\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " Oct 30 05:55:55.848674 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848443 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-podinfo\") pod \"117dc795-66d4-478c-9ff7-b23b056a4d48\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " Oct 30 05:55:55.848674 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848493 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-data\") pod \"117dc795-66d4-478c-9ff7-b23b056a4d48\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " Oct 30 05:55:55.848674 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848585 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-token\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-token\") pod \"117dc795-66d4-478c-9ff7-b23b056a4d48\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " Oct 30 05:55:55.848674 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848638 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-credential-socket\") pod \"82915e36-0e03-48c5-8d83-942037d0e5f5\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " Oct 30 05:55:55.848674 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848662 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-kube-api-access-pnvnc\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-kube-api-access-pnvnc\") pod \"117dc795-66d4-478c-9ff7-b23b056a4d48\" (UID: \"117dc795-66d4-478c-9ff7-b23b056a4d48\") " Oct 30 05:55:55.848674 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848703 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-podinfo\") pod \"82915e36-0e03-48c5-8d83-942037d0e5f5\" (UID: \"82915e36-0e03-48c5-8d83-942037d0e5f5\") " Oct 30 05:55:55.849207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848757 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-certs" pod "82915e36-0e03-48c5-8d83-942037d0e5f5" (UID: "82915e36-0e03-48c5-8d83-942037d0e5f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.849207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848771 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-certs" pod "117dc795-66d4-478c-9ff7-b23b056a4d48" (UID: "117dc795-66d4-478c-9ff7-b23b056a4d48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.849207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.848969 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-credential-socket" pod "117dc795-66d4-478c-9ff7-b23b056a4d48" (UID: "117dc795-66d4-478c-9ff7-b23b056a4d48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.849207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.849090 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82915e36-0e03-48c5-8d83-942037d0e5f5-istiod-ca-cert" pod "82915e36-0e03-48c5-8d83-942037d0e5f5" (UID: "82915e36-0e03-48c5-8d83-942037d0e5f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.849732 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.849423 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117dc795-66d4-478c-9ff7-b23b056a4d48-istiod-ca-cert" pod "117dc795-66d4-478c-9ff7-b23b056a4d48" (UID: "117dc795-66d4-478c-9ff7-b23b056a4d48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850067 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850093 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850106 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850122 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850134 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82915e36-0e03-48c5-8d83-942037d0e5f5-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850147 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850161 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850174 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117dc795-66d4-478c-9ff7-b23b056a4d48-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850103 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-credential-socket" pod "82915e36-0e03-48c5-8d83-942037d0e5f5" (UID: "82915e36-0e03-48c5-8d83-942037d0e5f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850188 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850157 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-data" pod "117dc795-66d4-478c-9ff7-b23b056a4d48" (UID: "117dc795-66d4-478c-9ff7-b23b056a4d48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850201 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850216 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5xtn\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-kube-api-access-g5xtn\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850230 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850242 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850254 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850267 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac75576-ed42-464e-b4ce-b7659ff9c7e1-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.850298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.850279 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.853222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.853203 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:55:55.853949 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.853921 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-token" pod "82915e36-0e03-48c5-8d83-942037d0e5f5" (UID: "82915e36-0e03-48c5-8d83-942037d0e5f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.854016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.854001 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-kube-api-access-q2fhl" pod "82915e36-0e03-48c5-8d83-942037d0e5f5" (UID: "82915e36-0e03-48c5-8d83-942037d0e5f5"). InnerVolumeSpecName "kube-api-access-q2fhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.855410 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.855382 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-podinfo" pod "82915e36-0e03-48c5-8d83-942037d0e5f5" (UID: "82915e36-0e03-48c5-8d83-942037d0e5f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.855562 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.855394 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-envoy" pod "117dc795-66d4-478c-9ff7-b23b056a4d48" (UID: "117dc795-66d4-478c-9ff7-b23b056a4d48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.855722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.855701 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-token" pod "117dc795-66d4-478c-9ff7-b23b056a4d48" (UID: "117dc795-66d4-478c-9ff7-b23b056a4d48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.857630 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.857607 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-podinfo" pod "117dc795-66d4-478c-9ff7-b23b056a4d48" (UID: "117dc795-66d4-478c-9ff7-b23b056a4d48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:55.860404 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.860376 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-envoy" pod "82915e36-0e03-48c5-8d83-942037d0e5f5" (UID: "82915e36-0e03-48c5-8d83-942037d0e5f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:55.861654 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.861629 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-kube-api-access-pnvnc" pod "117dc795-66d4-478c-9ff7-b23b056a4d48" (UID: "117dc795-66d4-478c-9ff7-b23b056a4d48"). InnerVolumeSpecName "kube-api-access-pnvnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.866758 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.866738 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w"] Oct 30 05:55:55.871769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.871749 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-rdm7w"] Oct 30 05:55:55.923997 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.923907 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx"] Oct 30 05:55:55.928540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.928515 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-6zxqx"] Oct 30 05:55:55.951688 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.951619 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829"] Oct 30 05:55:55.952821 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952798 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.952821 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952824 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.953086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952837 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.953086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952850 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2fhl\" (UniqueName: \"kubernetes.io/projected/82915e36-0e03-48c5-8d83-942037d0e5f5-kube-api-access-q2fhl\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.953086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952864 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.953086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952877 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.953086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952889 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.953086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952901 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82915e36-0e03-48c5-8d83-942037d0e5f5-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.953086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952927 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnvnc\" (UniqueName: \"kubernetes.io/projected/117dc795-66d4-478c-9ff7-b23b056a4d48-kube-api-access-pnvnc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.953086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.952940 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82915e36-0e03-48c5-8d83-942037d0e5f5-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.957768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.957747 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-dr829"] Oct 30 05:55:55.987351 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.984561 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc"] Oct 30 05:55:55.989494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:55.989428 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-vqjbc"] Oct 30 05:55:56.114312 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.114280 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5"] Oct 30 05:55:56.124158 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.124131 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-txfb5"] Oct 30 05:55:56.155962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.155891 5072 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:55:56.156135 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.156008 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:56.165766 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.165732 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc"] Oct 30 05:55:56.169976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.169952 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-hlwzc"] Oct 30 05:55:56.526887 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.526787 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" path="/var/lib/kubelet/pods/1e81f2fe-42c9-48c2-a934-6d45733a369f/volumes" Oct 30 05:55:56.527620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.527598 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43331d15-6de3-4311-995b-30953745e472" path="/var/lib/kubelet/pods/43331d15-6de3-4311-995b-30953745e472/volumes" Oct 30 05:55:56.528301 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.528281 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" path="/var/lib/kubelet/pods/494ff58d-8474-487f-8bbb-a7d068c56a65/volumes" Oct 30 05:55:56.529448 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.529416 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" path="/var/lib/kubelet/pods/8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad/volumes" Oct 30 05:55:56.530109 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.530089 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" path="/var/lib/kubelet/pods/cb62c6d0-9478-4a3e-b43b-9731333b9db4/volumes" Oct 30 05:55:56.530759 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.530737 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" path="/var/lib/kubelet/pods/e7caf44c-c7f8-4daa-8957-1ea7b2729676/volumes" Oct 30 05:55:56.531832 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.531811 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" path="/var/lib/kubelet/pods/f05e3600-3aa0-4c56-97ec-5fa829f18c7e/volumes" Oct 30 05:55:56.534111 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.534088 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd3ba4a-8402-4f98-9d16-b51b61baf453" path="/var/lib/kubelet/pods/fdd3ba4a-8402-4f98-9d16-b51b61baf453/volumes" Oct 30 05:55:56.702795 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.702767 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" Oct 30 05:55:56.703055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.702788 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn" event={"ID":"117dc795-66d4-478c-9ff7-b23b056a4d48","Type":"ContainerDied","Data":"a4a4840eed66dd32d287e64761391aa167628920349b26b1dccef826db13f805"} Oct 30 05:55:56.703209 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.703130 5072 scope.go:122] "RemoveContainer" containerID="71e21cfe4e48d19b84e0ad35614974e470ce0d978bf8894cb987bc5aa3801355" Oct 30 05:55:56.710317 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.710282 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" event={"ID":"0e382939-b971-4260-8c87-fcdad8ec96c2","Type":"ContainerStarted","Data":"2389843bd1696cc3e3440211361265aee3e0b6729cbf969d306f7fcd6a2736d1"} Oct 30 05:55:56.710531 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.710346 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" event={"ID":"0e382939-b971-4260-8c87-fcdad8ec96c2","Type":"ContainerStarted","Data":"d6af83c08c2aaf7c4aa33752fefe41b4dd0c2c035853fd93935260a5f77be5f4"} Oct 30 05:55:56.711288 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.711235 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:55:56.715201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.715179 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" event={"ID":"82915e36-0e03-48c5-8d83-942037d0e5f5","Type":"ContainerDied","Data":"228fe673fda5e87b699d520038dc8e2b2dbbf3d6a3c32daf0a5609b141728586"} Oct 30 05:55:56.715303 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.715283 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42" Oct 30 05:55:56.719866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.719843 5072 scope.go:122] "RemoveContainer" containerID="6d84558119fb5bc081798d9d481fc59cdf7ec223afe177983a954e7873a58de0" Oct 30 05:55:56.724987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.724957 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" event={"ID":"af170278-15d4-44d0-9f2c-3a91ad682991","Type":"ContainerStarted","Data":"30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88"} Oct 30 05:55:56.725097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.725016 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" event={"ID":"af170278-15d4-44d0-9f2c-3a91ad682991","Type":"ContainerStarted","Data":"96f9db896d69b8142c1049982004e195a0e5186fc2538994c22b18bbdabf07a9"} Oct 30 05:55:56.725397 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.725380 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4" Oct 30 05:55:56.725902 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.725871 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:55:56.772914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.772881 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn"] Oct 30 05:55:56.778648 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.778587 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-bxkpn"] Oct 30 05:55:56.829188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.829133 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" podStartSLOduration=14.829114658 podStartE2EDuration="14.829114658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:56.822660352 +0000 UTC m=+1516.426349621" watchObservedRunningTime="2026-10-30 05:55:56.829114658 +0000 UTC m=+1516.432804224" Oct 30 05:55:56.868175 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.868136 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4"] Oct 30 05:55:56.891698 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:56.891668 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-hjzv4"] Oct 30 05:55:57.031688 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:57.031597 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42"] Oct 30 05:55:57.042028 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:57.041996 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-9dj42"] Oct 30 05:55:57.046520 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:57.046362 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" podStartSLOduration=8.693470063 podStartE2EDuration="9.046347279s" totalImagesPullingTime="352.877216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:57.030215539 +0000 UTC m=+1516.633904799" watchObservedRunningTime="2026-10-30 05:55:57.046347279 +0000 UTC m=+1516.650036535" Oct 30 05:55:58.526712 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:58.525806 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" path="/var/lib/kubelet/pods/117dc795-66d4-478c-9ff7-b23b056a4d48/volumes" Oct 30 05:55:58.526712 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:58.526690 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" path="/var/lib/kubelet/pods/3ac75576-ed42-464e-b4ce-b7659ff9c7e1/volumes" Oct 30 05:55:58.527847 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:55:58.527423 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" path="/var/lib/kubelet/pods/82915e36-0e03-48c5-8d83-942037d0e5f5/volumes" Oct 30 05:56:00.717728 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:56:00.717686 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:56:02.354016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:02.353984 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:56:03.276917 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.276876 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft"] Oct 30 05:56:03.277670 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277654 5072 cpu_manager.go:414] "containerMap: removing container" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" Oct 30 05:56:03.277711 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277672 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" Oct 30 05:56:03.277711 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277691 5072 cpu_manager.go:414] "containerMap: removing container" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" Oct 30 05:56:03.277711 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277696 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" Oct 30 05:56:03.277711 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277705 5072 cpu_manager.go:414] "containerMap: removing container" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" Oct 30 05:56:03.277711 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277711 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277720 5072 cpu_manager.go:414] "containerMap: removing container" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277724 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277733 5072 cpu_manager.go:414] "containerMap: removing container" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277738 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277745 5072 cpu_manager.go:414] "containerMap: removing container" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277748 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277759 5072 cpu_manager.go:414] "containerMap: removing container" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277763 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277775 5072 cpu_manager.go:414] "containerMap: removing container" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277779 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277793 5072 cpu_manager.go:414] "containerMap: removing container" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277798 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277817 5072 cpu_manager.go:414] "containerMap: removing container" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" Oct 30 05:56:03.277860 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277821 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277931 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="82915e36-0e03-48c5-8d83-942037d0e5f5" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277939 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="43331d15-6de3-4311-995b-30953745e472" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277947 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac75576-ed42-464e-b4ce-b7659ff9c7e1" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277953 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05e3600-3aa0-4c56-97ec-5fa829f18c7e" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277960 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277965 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="117dc795-66d4-478c-9ff7-b23b056a4d48" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277974 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb62c6d0-9478-4a3e-b43b-9731333b9db4" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277983 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe8a4c8-f5f3-4dcb-a3a9-e9e8ef2723ad" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277990 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.277996 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7caf44c-c7f8-4daa-8957-1ea7b2729676" containerName="istio-proxy" Oct 30 05:56:03.278251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.278002 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e81f2fe-42c9-48c2-a934-6d45733a369f" containerName="istio-proxy" Oct 30 05:56:03.281856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.281841 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.303864 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.303828 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft"] Oct 30 05:56:03.308438 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.308412 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-530a5e-serving-cert\"" Oct 30 05:56:03.413482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.409855 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-metrics-certs\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.413482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.409953 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-maas-api-tls\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.413482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.410054 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6fp7\" (UniqueName: \"kubernetes.io/projected/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-kube-api-access-b6fp7\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.413482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.410109 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.526477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.526407 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-metrics-certs\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.526833 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.526814 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-maas-api-tls\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.527260 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.527195 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fp7\" (UniqueName: \"kubernetes.io/projected/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-kube-api-access-b6fp7\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.528496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.528246 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.546499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.543147 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-metrics-certs\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.546976 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.546942 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.553510 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.548009 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-maas-api-tls\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.559965 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.559937 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fp7\" (UniqueName: \"kubernetes.io/projected/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-kube-api-access-b6fp7\") pod \"maas-api-e2e-api-b-530a5e-795dfd6786-bwvft\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.596639 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.596608 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:03.756374 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.756120 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 05:56:03.911928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:03.911892 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft"] Oct 30 05:56:03.912603 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:56:03.912576 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d5902b_5ff6_4217_a48e_33a0bf5056aa.slice/crio-cd2369fcc594b45cb01c06c363f42b676b294e07ebca07b53194a8a452947544 WatchSource:0}: Error finding container cd2369fcc594b45cb01c06c363f42b676b294e07ebca07b53194a8a452947544: Status 404 returned error can't find the container with id cd2369fcc594b45cb01c06c363f42b676b294e07ebca07b53194a8a452947544 Oct 30 05:56:04.804597 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:04.804495 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" event={"ID":"f2d5902b-5ff6-4217-a48e-33a0bf5056aa","Type":"ContainerStarted","Data":"2915c29dbf333c8929fb9c84575bd9f0bae05a202cd312200027be9cd277aa2d"} Oct 30 05:56:04.804597 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:04.804565 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" event={"ID":"f2d5902b-5ff6-4217-a48e-33a0bf5056aa","Type":"ContainerStarted","Data":"cd2369fcc594b45cb01c06c363f42b676b294e07ebca07b53194a8a452947544"} Oct 30 05:56:04.804597 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:04.804594 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:04.847296 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:04.847245 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" podStartSLOduration=1.384399455 podStartE2EDuration="1.847227447s" totalImagesPullingTime="462.827992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:04.8441152 +0000 UTC m=+1524.447804457" watchObservedRunningTime="2026-10-30 05:56:04.847227447 +0000 UTC m=+1524.450916707" Oct 30 05:56:08.785027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:08.784993 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:56:10.845684 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:10.845647 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:56:12.298106 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.298073 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk"] Oct 30 05:56:12.298847 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.298833 5072 cpu_manager.go:414] "containerMap: removing container" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" Oct 30 05:56:12.298906 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.298849 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494ff58d-8474-487f-8bbb-a7d068c56a65" containerName="istio-proxy" Oct 30 05:56:12.304042 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.304024 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.334882 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.334851 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-3abe42-serving-cert\"" Oct 30 05:56:12.336755 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.336722 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk"] Oct 30 05:56:12.370690 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.370659 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.370969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.370944 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0359cfa7-f684-476e-a9e6-a2defd1feb2b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.371036 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.370997 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fl9\" (UniqueName: \"kubernetes.io/projected/0359cfa7-f684-476e-a9e6-a2defd1feb2b-kube-api-access-x7fl9\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.371122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.371105 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.472832 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.472797 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0359cfa7-f684-476e-a9e6-a2defd1feb2b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.472832 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.472836 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fl9\" (UniqueName: \"kubernetes.io/projected/0359cfa7-f684-476e-a9e6-a2defd1feb2b-kube-api-access-x7fl9\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.473097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.472881 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.473097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.473052 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.473718 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.473693 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0359cfa7-f684-476e-a9e6-a2defd1feb2b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.475655 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.475633 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.475776 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.475674 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.496826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.496797 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fl9\" (UniqueName: \"kubernetes.io/projected/0359cfa7-f684-476e-a9e6-a2defd1feb2b-kube-api-access-x7fl9\") pod \"maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:12.620896 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:12.620860 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:13.006286 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:13.006260 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk"] Oct 30 05:56:13.006937 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:56:13.006908 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0359cfa7_f684_476e_a9e6_a2defd1feb2b.slice/crio-40996a75c3a58e8ef718edcac51f09d33a3896b0d2221ca981d4c0ce01224e89 WatchSource:0}: Error finding container 40996a75c3a58e8ef718edcac51f09d33a3896b0d2221ca981d4c0ce01224e89: Status 404 returned error can't find the container with id 40996a75c3a58e8ef718edcac51f09d33a3896b0d2221ca981d4c0ce01224e89 Oct 30 05:56:13.032018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:13.031960 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" event={"ID":"0359cfa7-f684-476e-a9e6-a2defd1feb2b","Type":"ContainerStarted","Data":"40996a75c3a58e8ef718edcac51f09d33a3896b0d2221ca981d4c0ce01224e89"} Oct 30 05:56:14.042222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.042187 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" event={"ID":"0359cfa7-f684-476e-a9e6-a2defd1feb2b","Type":"ContainerStarted","Data":"2de5f0e5086be1d4a2707b4699d43a28c205803de7998383367baea58de773b1"} Oct 30 05:56:14.042960 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.042398 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:14.086501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.086424 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" podStartSLOduration=1.740109823 podStartE2EDuration="2.086408908s" totalImagesPullingTime="346.299085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:14.081878846 +0000 UTC m=+1533.685568103" watchObservedRunningTime="2026-10-30 05:56:14.086408908 +0000 UTC m=+1533.690098165" Oct 30 05:56:14.129930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.129891 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz"] Oct 30 05:56:14.135839 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.135815 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.159495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.159444 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz"] Oct 30 05:56:14.179571 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.179544 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4323f6-serving-cert\"" Oct 30 05:56:14.251950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.251908 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423923f-15e4-4204-86de-5a6483ee8356-openshift-service-ca\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.252187 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.252156 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-maas-api-tls\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.252339 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.252319 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq2nz\" (UniqueName: \"kubernetes.io/projected/2423923f-15e4-4204-86de-5a6483ee8356-kube-api-access-fq2nz\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.252443 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.252429 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-metrics-certs\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.393738 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.393284 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-maas-api-tls\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.393738 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.393493 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq2nz\" (UniqueName: \"kubernetes.io/projected/2423923f-15e4-4204-86de-5a6483ee8356-kube-api-access-fq2nz\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.393738 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.393623 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-metrics-certs\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.393738 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.393708 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423923f-15e4-4204-86de-5a6483ee8356-openshift-service-ca\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.395488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.394657 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423923f-15e4-4204-86de-5a6483ee8356-openshift-service-ca\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.405342 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.405314 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-metrics-certs\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.407486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.407434 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-maas-api-tls\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.447534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.444371 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq2nz\" (UniqueName: \"kubernetes.io/projected/2423923f-15e4-4204-86de-5a6483ee8356-kube-api-access-fq2nz\") pod \"maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.469847 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.469756 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:14.846336 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:56:14.846285 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2423923f_15e4_4204_86de_5a6483ee8356.slice/crio-04703b4bfc88cd19643f44783cc6a7d2b995b6345bd4f5afb0a810e01fa7a457 WatchSource:0}: Error finding container 04703b4bfc88cd19643f44783cc6a7d2b995b6345bd4f5afb0a810e01fa7a457: Status 404 returned error can't find the container with id 04703b4bfc88cd19643f44783cc6a7d2b995b6345bd4f5afb0a810e01fa7a457 Oct 30 05:56:14.847660 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:14.847633 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz"] Oct 30 05:56:15.051321 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:15.051285 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" event={"ID":"2423923f-15e4-4204-86de-5a6483ee8356","Type":"ContainerStarted","Data":"04703b4bfc88cd19643f44783cc6a7d2b995b6345bd4f5afb0a810e01fa7a457"} Oct 30 05:56:16.061063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:16.061030 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" event={"ID":"2423923f-15e4-4204-86de-5a6483ee8356","Type":"ContainerStarted","Data":"e582570eff99ba2387b5a242acc45524dce24450dba0d1535b231d7741338046"} Oct 30 05:56:16.061491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:16.061115 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:16.131534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:16.124643 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" podStartSLOduration=1.74870952 podStartE2EDuration="2.124621634s" totalImagesPullingTime="375.912114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:16.121269665 +0000 UTC m=+1535.724958923" watchObservedRunningTime="2026-10-30 05:56:16.124621634 +0000 UTC m=+1535.728310916" Oct 30 05:56:17.179700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.179667 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f"] Oct 30 05:56:17.185192 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.185168 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.202851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.202827 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq"] Oct 30 05:56:17.207861 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.207831 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.214120 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.214097 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d"] Oct 30 05:56:17.220045 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.220025 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.220651 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.220581 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f"] Oct 30 05:56:17.238447 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.237154 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n"] Oct 30 05:56:17.262208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.261145 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.269415 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269388 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d"] Oct 30 05:56:17.269666 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269647 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.269755 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269695 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.269755 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269727 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.269755 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269749 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.269930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269801 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.269930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269824 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.269930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269856 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.269930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269876 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.269930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269905 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.269930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.269919 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.270250 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.270002 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.270250 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.270195 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5gr5\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-kube-api-access-l5gr5\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.270250 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.270233 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.270413 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.270260 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.270413 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.270306 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.270413 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.270372 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.270413 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.270393 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8mj\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-kube-api-access-mt8mj\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.270644 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.270434 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.293487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.292202 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq"] Oct 30 05:56:17.298610 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.296652 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n"] Oct 30 05:56:17.371857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.371824 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.372071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.371888 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.372071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.371913 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.372071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.371946 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.372071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.371979 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.372071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372003 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7bp\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-kube-api-access-wx7bp\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.372071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372058 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.372526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372093 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.372526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372208 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.372526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372289 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.372526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372332 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.372526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372339 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.372526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372362 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.372526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372368 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.372526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372379 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.372526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372525 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.373001 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372579 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.373001 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.372774 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.373156 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373011 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.373334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373312 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.373410 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373377 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.373488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373424 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gr5\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-kube-api-access-l5gr5\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.373488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373471 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.373605 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373579 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.373667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373609 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.373667 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373626 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.373782 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373675 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.373782 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373747 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.373888 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373777 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b09b1b-4429-4301-be01-fa1b1ad0670a-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.373888 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373807 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.373888 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373846 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.374053 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373913 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.374053 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373946 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.374053 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373965 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8mj\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-kube-api-access-mt8mj\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.374053 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.373990 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmvxn\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-kube-api-access-cmvxn\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.374053 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.374018 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.374059 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.374187 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.374208 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.374230 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.374259 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.374281 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.375730 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.375988 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.376622 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.377238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.376848 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.377806 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.377768 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.378229 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.378205 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.378535 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.378511 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.379801 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.379777 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.395720 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.395692 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.397451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.397428 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gr5\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-kube-api-access-l5gr5\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.400956 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.400924 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8mj\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-kube-api-access-mt8mj\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-llr7f\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.401442 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.401426 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-pzvfq\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.476281 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476198 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.476281 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476243 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.476281 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476272 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.476643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476295 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.476643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476380 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.476643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476413 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.476643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476435 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7bp\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-kube-api-access-wx7bp\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.476643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476567 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.476643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476609 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.476643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476636 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476710 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476805 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476804 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476815 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476837 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b09b1b-4429-4301-be01-fa1b1ad0670a-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476839 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476866 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476904 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476944 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476989 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmvxn\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-kube-api-access-cmvxn\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477562 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.477023 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.477562 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.477024 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477562 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.476987 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.477562 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.477277 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477562 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.477485 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.477833 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.477749 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.477884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.477847 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.478446 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.478410 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b09b1b-4429-4301-be01-fa1b1ad0670a-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.479410 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.479386 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.479630 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.479584 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.479921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.479904 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.479980 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.479966 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.498442 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.498415 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7bp\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-kube-api-access-wx7bp\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.498599 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.498587 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.498696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.498676 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8592n\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.500672 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.500651 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:17.503584 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.503565 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmvxn\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-kube-api-access-cmvxn\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-zz67d\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.528320 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.528293 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:17.595112 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.595060 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:17.609893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.609629 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:17.844505 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.844263 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f"] Oct 30 05:56:17.848164 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.848130 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:17.848282 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.848216 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:17.848282 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.848266 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:17.874108 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.874083 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq"] Oct 30 05:56:17.874250 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:56:17.874220 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee667dc4_ddc3_49c6_b87f_02bc14d31a6c.slice/crio-d23b7df6de0212997d6d49d79cc28d8a0728011164b3a894ca80be93107ccf0f WatchSource:0}: Error finding container d23b7df6de0212997d6d49d79cc28d8a0728011164b3a894ca80be93107ccf0f: Status 404 returned error can't find the container with id d23b7df6de0212997d6d49d79cc28d8a0728011164b3a894ca80be93107ccf0f Oct 30 05:56:17.878737 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.877609 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:17.878737 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.877695 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:17.878737 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.877738 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:17.912473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.912094 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl"] Oct 30 05:56:17.923716 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.923691 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:17.964246 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.939301 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-76b905-serving-cert\"" Oct 30 05:56:17.970919 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.970819 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl"] Oct 30 05:56:17.983994 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.983861 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d"] Oct 30 05:56:17.987011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.986979 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:17.987211 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.987188 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:17.987361 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.987342 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:17.996583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.995119 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2hgv\" (UniqueName: \"kubernetes.io/projected/2f5ad48b-980d-4740-aa22-2af1faa69ab5-kube-api-access-q2hgv\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:17.996583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.995238 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5ad48b-980d-4740-aa22-2af1faa69ab5-openshift-service-ca\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:17.996583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.995276 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-metrics-certs\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:17.996583 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:17.995371 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-maas-api-tls\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.009217 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.008452 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:18.009217 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.008556 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:18.009217 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.008597 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:18.016315 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.016293 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n"] Oct 30 05:56:18.098804 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.098769 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" event={"ID":"dbeaf7e5-398d-4ef7-a2b1-310d653279a2","Type":"ContainerStarted","Data":"e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13"} Oct 30 05:56:18.098978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.098836 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" event={"ID":"dbeaf7e5-398d-4ef7-a2b1-310d653279a2","Type":"ContainerStarted","Data":"3bdb1d70a141948aaf213b02db73c34721061e3635a51cd498c8759a4e82c7b3"} Oct 30 05:56:18.102063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.102029 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" event={"ID":"0f42b70a-73d5-4f6c-9b09-afe535fda7f5","Type":"ContainerStarted","Data":"8f4839c654ebb2e1f5b3f437071b9f189af0fde6bffa8bb1af097f456862855e"} Oct 30 05:56:18.104795 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.104764 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5ad48b-980d-4740-aa22-2af1faa69ab5-openshift-service-ca\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.104949 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.104817 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-metrics-certs\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.104949 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.104911 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-maas-api-tls\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.105201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.105180 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2hgv\" (UniqueName: \"kubernetes.io/projected/2f5ad48b-980d-4740-aa22-2af1faa69ab5-kube-api-access-q2hgv\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.106405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.106376 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5ad48b-980d-4740-aa22-2af1faa69ab5-openshift-service-ca\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.107550 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.107451 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" event={"ID":"02b09b1b-4429-4301-be01-fa1b1ad0670a","Type":"ContainerStarted","Data":"eb759b58d8d0aa2a14d3fe14b2328974f3e0fdb6f6536c555754f3757b0db751"} Oct 30 05:56:18.109731 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.109685 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" event={"ID":"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c","Type":"ContainerStarted","Data":"8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404"} Oct 30 05:56:18.109902 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.109742 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" event={"ID":"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c","Type":"ContainerStarted","Data":"d23b7df6de0212997d6d49d79cc28d8a0728011164b3a894ca80be93107ccf0f"} Oct 30 05:56:18.114569 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.114542 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-maas-api-tls\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.120595 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.120578 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-metrics-certs\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.215019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.214972 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podStartSLOduration=1.214955187 podStartE2EDuration="1.214955187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:18.173031763 +0000 UTC m=+1537.776721021" watchObservedRunningTime="2026-10-30 05:56:18.214955187 +0000 UTC m=+1537.818644443" Oct 30 05:56:18.231437 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.231342 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podStartSLOduration=1.2313263110000001 podStartE2EDuration="1.231326311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:18.227543959 +0000 UTC m=+1537.831233243" watchObservedRunningTime="2026-10-30 05:56:18.231326311 +0000 UTC m=+1537.835015567" Oct 30 05:56:18.311795 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.304155 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2hgv\" (UniqueName: \"kubernetes.io/projected/2f5ad48b-980d-4740-aa22-2af1faa69ab5-kube-api-access-q2hgv\") pod \"maas-api-e2e-api-b-76b905-686f766f84-zjzpl\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.312700 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.312406 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:18.503531 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.501315 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:18.541049 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.541019 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:18.541743 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.541713 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:18.556435 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.556414 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:18.790245 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:18.790205 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl"] Oct 30 05:56:18.791655 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:56:18.791629 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5ad48b_980d_4740_aa22_2af1faa69ab5.slice/crio-fbfc51e09ba2cada69b5792e0178dc88fe4b7fd5df99f70110188521f9e28c2f WatchSource:0}: Error finding container fbfc51e09ba2cada69b5792e0178dc88fe4b7fd5df99f70110188521f9e28c2f: Status 404 returned error can't find the container with id fbfc51e09ba2cada69b5792e0178dc88fe4b7fd5df99f70110188521f9e28c2f Oct 30 05:56:19.189806 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.189767 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" event={"ID":"0f42b70a-73d5-4f6c-9b09-afe535fda7f5","Type":"ContainerStarted","Data":"bed074e8db8c6a68a16c2a085ad931896e80d7d72b9b1d776ab9363f66975cf3"} Oct 30 05:56:19.197991 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.197635 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" event={"ID":"02b09b1b-4429-4301-be01-fa1b1ad0670a","Type":"ContainerStarted","Data":"0a3da5637fe8cfe81d82b6c02cdd66e607e3941255d15e6bbd1fc191e3efe9e5"} Oct 30 05:56:19.200089 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.200063 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" event={"ID":"2f5ad48b-980d-4740-aa22-2af1faa69ab5","Type":"ContainerStarted","Data":"fbfc51e09ba2cada69b5792e0178dc88fe4b7fd5df99f70110188521f9e28c2f"} Oct 30 05:56:19.257484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.256649 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podStartSLOduration=2.256628455 podStartE2EDuration="2.256628455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:19.241114948 +0000 UTC m=+1538.844804200" watchObservedRunningTime="2026-10-30 05:56:19.256628455 +0000 UTC m=+1538.860317712" Oct 30 05:56:19.333545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.333478 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podStartSLOduration=2.333437023 podStartE2EDuration="2.333437023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:19.304901541 +0000 UTC m=+1538.908590799" watchObservedRunningTime="2026-10-30 05:56:19.333437023 +0000 UTC m=+1538.937126280" Oct 30 05:56:19.548947 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.548856 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:19.549216 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.549185 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:19.549825 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.549800 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:19.557545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.557274 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:19.603066 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.603036 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:19.611795 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.611766 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:19.626707 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.626668 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:19.663602 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.663558 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:19.665527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.626980 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:19.665714 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:19.665690 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:20.075181 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:20.073943 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 05:56:20.411485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:20.410890 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:20.638483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:20.613593 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:20.641443 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:20.641237 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:20.667608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:20.652660 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:20.667608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:20.652728 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:20.667608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:20.655614 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:20.667608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:20.655696 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:21.130147 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.129978 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:56:21.140576 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.132337 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:21.231486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.222523 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:21.260484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.246119 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:21.260484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.250057 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:21.260484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.250124 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:21.367364 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.365388 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 05:56:21.424910 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.424809 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:21.425577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.425498 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" event={"ID":"2f5ad48b-980d-4740-aa22-2af1faa69ab5","Type":"ContainerStarted","Data":"c3b9ef59040784a1671ceb69046ca3f747b4be79451aa093b16f82871f6dba72"} Oct 30 05:56:21.521055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.520986 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" podStartSLOduration=4.011370716 podStartE2EDuration="4.520962758s" totalImagesPullingTime="509.592042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:21.513412924 +0000 UTC m=+1541.117102182" watchObservedRunningTime="2026-10-30 05:56:21.520962758 +0000 UTC m=+1541.124652061" Oct 30 05:56:21.562959 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.562925 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:21.920940 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.920895 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:21.923293 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.643934 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:56:21.923694 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.598627 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:21.924222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.924194 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:21.924364 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.923873 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:56:21.925522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.618329 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:21.925738 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.925715 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:21.933573 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.619814 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:21.933734 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:21.933710 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:22.151769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.150057 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 05:56:22.151769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.150475 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:56:22.151769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.151582 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:22.152154 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.151909 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:56:22.234542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.206610 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:56:22.257485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.240669 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:22.257485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.241033 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:56:22.307484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.301140 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5"] Oct 30 05:56:22.432936 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.432895 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:22.566541 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.543792 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:56:22.567997 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.567966 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.656857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.656819 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:22.657162 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.657126 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:22.704486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.681728 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:22.704826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.704786 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:22.706587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.706560 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:22.706768 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.706741 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:22.850538 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.847159 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:22.857593 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.857549 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:22.868494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.858002 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.868819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.868787 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.876558 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.868984 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.941673 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.941633 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.942013 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.941988 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.942266 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.942240 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.952504 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.952448 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdqg\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-kube-api-access-ngdqg\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.952783 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.952762 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.952946 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.952927 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d986add-57ae-496e-a4d7-fdb2824f07a8-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:22.953300 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:22.929175 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:56:23.187636 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.176435 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:23.207638 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.207598 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.207829 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.207700 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.207829 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.207770 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.207938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.207871 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdqg\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-kube-api-access-ngdqg\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.207992 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.207942 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.208040 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.208020 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d986add-57ae-496e-a4d7-fdb2824f07a8-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.223487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.208357 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.223487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.208429 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.223487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.208532 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.249486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.233152 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.249486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.233245 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:23.249486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.241347 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.249486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.249479 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d986add-57ae-496e-a4d7-fdb2824f07a8-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.254984 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.254950 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.377752 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.376689 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.428129 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.428087 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:56:23.472064 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.461742 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d"] Oct 30 05:56:23.486695 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.468691 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 05:56:23.521950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.521909 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.623906 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.536007 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:23.624188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.624149 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:23.624321 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.571724 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5"] Oct 30 05:56:23.624987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.548736 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:23.648545 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.566645 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:23.648977 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.648942 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:23.649158 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.634151 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.649336 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.649318 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.667485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.657636 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.667485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.657879 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.667973 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.667945 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:23.668198 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.668159 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:23.668925 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.668902 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:23.669092 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.669071 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:23.742640 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.736828 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d"] Oct 30 05:56:23.862484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.837735 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:56:23.868697 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.829987 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdqg\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-kube-api-access-ngdqg\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.868933 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.848877 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.896636 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.858967 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.918848 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.918786 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:23.938529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.933093 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkwh\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-kube-api-access-bfkwh\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.938529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.933227 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.938529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.933274 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.938529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.933366 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.938529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.933483 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:23.963172 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.962546 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:23.986522 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.963533 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:56:24.021574 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:23.991574 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:56:24.048528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.025141 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:24.100992 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.100955 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkwh\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-kube-api-access-bfkwh\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.101296 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.101275 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.101433 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.101415 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.101648 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.101628 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.101826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.101809 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.102044 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.102027 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.102180 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.102162 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.102318 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.102302 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.102595 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.102576 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.110629 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.110605 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.131520 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.112087 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.148643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.124974 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.288888 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.288790 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.313510 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.312297 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkwh\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-kube-api-access-bfkwh\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.341588 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.341545 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:24.371873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.371838 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.372114 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.342661 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.372208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.370831 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.883485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.882619 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:24.947257 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.947222 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:24.948330 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.948291 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:24.948575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.948529 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:25.023719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.949119 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:25.023719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.023694 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:25.023970 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:24.977736 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:25.023970 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.023768 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:25.033239 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.032961 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:25.320521 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.045817 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:56:25.324481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.060902 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:25.380527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.328535 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:25.400635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.114313 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:56:25.403319 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.114875 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:56:25.404164 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.113703 5072 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod157a8419-22b9-4588-870a-0245bbefbf39"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod157a8419-22b9-4588-870a-0245bbefbf39] : Timed out while waiting for systemd to remove kubepods-burstable-pod157a8419_22b9_4588_870a_0245bbefbf39.slice" Oct 30 05:56:25.404397 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:56:25.404335 5072 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod157a8419-22b9-4588-870a-0245bbefbf39] : unable to destroy cgroup paths for cgroup [kubepods burstable pod157a8419-22b9-4588-870a-0245bbefbf39] : Timed out while waiting for systemd to remove kubepods-burstable-pod157a8419_22b9_4588_870a_0245bbefbf39.slice" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" podUID="157a8419-22b9-4588-870a-0245bbefbf39" Oct 30 05:56:25.410904 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.407078 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:25.501431 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.501396 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:25.501742 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.501696 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:25.557933 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.557832 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:25.558845 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.558803 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:25.631488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.605998 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:25.632628 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.632584 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:25.649007 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.639153 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:25.649269 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:25.649236 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:26.187028 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.147549 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 05:56:26.461481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.425430 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6" Oct 30 05:56:26.494488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.456058 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:26.494488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.456096 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 05:56:26.494488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.492438 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:56:26.494488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.492562 5072 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:56:26.494488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.456792 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:26.741560 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.730822 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:26.741560 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.730875 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:26.746496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.744573 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:26.746496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.744626 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:26.746496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.744702 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:26.746496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.744723 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:26.792540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.791424 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:26.792540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:26.791506 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:27.154956 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.154915 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:27.155566 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.155348 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:27.505324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.505281 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:27.779699 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.532054 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:27.825509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.538731 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:27.825509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.821490 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:27.825509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.538845 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:27.825509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.821664 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:27.825509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.598223 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:27.825509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.821800 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:27.859798 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.859762 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:27.860086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.860056 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:27.860238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.777722 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:27.900162 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.900103 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:28.040542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.012270 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:28.040542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.012375 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:28.040542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.012419 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:28.106483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.101570 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:28.108123 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:27.905267 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:28.108510 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.108433 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:28.267484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.266539 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:28.337565 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.289737 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:56:28.522843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.522792 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:28.523474 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.523415 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:28.581804 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.581768 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:28.615341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.613618 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:28.615341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.614708 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:28.615341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.614749 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:28.615341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.615061 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:28.615341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.615095 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:28.650515 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:56:28.645883 5072 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.917s" Oct 30 05:56:28.650515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.645929 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" event={"ID":"9d986add-57ae-496e-a4d7-fdb2824f07a8","Type":"ContainerStarted","Data":"ba2ae62a4a8cf1524b38abf8242cf12d3d9996afc0d451637be2a24fd7b066e9"} Oct 30 05:56:28.829049 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.827392 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:28.888554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.864104 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:28.888554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.887921 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:56:28.888554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.887978 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:56:28.888554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.888024 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 05:56:28.888554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.888065 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:56:28.888554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.888110 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5"] Oct 30 05:56:28.888554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.888143 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6"] Oct 30 05:56:28.888554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.888165 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-xrwk6"] Oct 30 05:56:28.888554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:28.888201 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d"] Oct 30 05:56:29.530546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.525310 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 05:56:29.606035 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.564853 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 05:56:29.606874 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.593087 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 05:56:29.607059 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.607025 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 05:56:29.639747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.636871 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 05:56:29.639747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.638134 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 05:56:29.639747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.638199 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 05:56:29.658061 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.658026 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 05:56:29.682517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.677545 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 05:56:29.682517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.680661 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 05:56:29.682517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.680721 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 05:56:29.684436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.683068 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:56:29.684436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.683132 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 05:56:29.776265 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.774654 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" event={"ID":"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7","Type":"ContainerStarted","Data":"c04f3cb34d556ecc1ec1e4d3a6ab09f7dcc678a1f6ddacbc29cf5853c8cc97b1"} Oct 30 05:56:29.849332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.849174 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" event={"ID":"9d986add-57ae-496e-a4d7-fdb2824f07a8","Type":"ContainerStarted","Data":"28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d"} Oct 30 05:56:29.849730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.849699 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:56:29.879909 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:29.879863 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:30.067414 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.067373 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:30.069832 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.067855 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:56:30.069832 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.067919 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:56:30.114219 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.114149 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podStartSLOduration=9.114125807 podStartE2EDuration="9.114125807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:29.927895313 +0000 UTC m=+1549.531584581" watchObservedRunningTime="2026-10-30 05:56:30.114125807 +0000 UTC m=+1549.717815122" Oct 30 05:56:30.129480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.128024 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 05:56:30.508071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.507979 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:30.508783 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.508709 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:30.509623 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.509604 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:56:30.556382 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.556341 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157a8419-22b9-4588-870a-0245bbefbf39" path="/var/lib/kubelet/pods/157a8419-22b9-4588-870a-0245bbefbf39/volumes" Oct 30 05:56:30.556939 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.556919 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:30.557629 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.557602 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:30.558450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.558428 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:56:30.603451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.603421 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:30.604896 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.604877 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:30.617597 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.605828 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:56:30.688214 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.688183 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:30.696233 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.696207 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:30.698427 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:30.698403 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:56:31.086875 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:56:31.086835 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:56:31.100193 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:31.100134 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:56:31.100362 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:31.100205 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:56:31.490740 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:31.489635 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" event={"ID":"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7","Type":"ContainerStarted","Data":"f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4"} Oct 30 05:56:31.659486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:31.658410 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" podStartSLOduration=10.658387652 podStartE2EDuration="10.658387652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:31.632112833 +0000 UTC m=+1551.235802091" watchObservedRunningTime="2026-10-30 05:56:31.658387652 +0000 UTC m=+1551.262076910" Oct 30 05:56:32.061485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:32.054754 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:32.071578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:32.061811 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:56:32.071790 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:32.063960 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:56:32.071962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:32.071935 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:56:32.072086 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:32.072011 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:56:33.315657 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:33.315618 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:56:33.391642 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:33.391599 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:56:33.391895 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:33.317011 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:56:33.392013 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:33.391992 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:56:33.697532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:33.697500 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq"] Oct 30 05:56:33.697999 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:33.697966 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" containerID="cri-o://ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686" gracePeriod=30 Oct 30 05:56:33.739877 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:33.703690 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:33.966884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:33.966788 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 05:56:34.098080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:34.098036 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6"] Oct 30 05:56:34.098755 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:34.098716 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" containerID="cri-o://83c6a223383342c8172709a641a524087332d6c65a97330203d813281fe10014" gracePeriod=30 Oct 30 05:56:34.104542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:34.104498 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:56:34.104730 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:34.104702 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:56:34.129479 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:34.107091 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:56:34.129479 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:34.107133 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:56:35.106926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:35.086111 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:56:35.106926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:35.086185 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:56:35.106926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:35.092550 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:56:35.106926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:35.092599 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:56:36.049704 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:36.049669 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:56:36.049985 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:36.049958 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:56:36.069912 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:36.069882 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:56:36.070129 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:36.070101 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:56:37.064484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:37.055523 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 05:56:37.064484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:37.055589 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 05:56:37.068568 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:37.068070 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 05:56:37.068568 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:37.068126 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 05:56:38.071669 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:38.071635 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:38.072125 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:38.072096 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:38.072944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:38.072920 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:56:38.097657 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:38.097631 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:38.097925 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:38.097903 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:38.098807 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:38.098786 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:56:39.093229 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.093204 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:56:39.158103 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.158071 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-envoy\") pod \"c9186dda-184a-47f6-913e-e3ba748da94d\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " Oct 30 05:56:39.158231 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.158217 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9186dda-184a-47f6-913e-e3ba748da94d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9186dda-184a-47f6-913e-e3ba748da94d-istio-podinfo\") pod \"c9186dda-184a-47f6-913e-e3ba748da94d\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " Oct 30 05:56:39.158290 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.158279 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-certs\") pod \"c9186dda-184a-47f6-913e-e3ba748da94d\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " Oct 30 05:56:39.158347 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.158312 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-data\") pod \"c9186dda-184a-47f6-913e-e3ba748da94d\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " Oct 30 05:56:39.158532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.158408 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-socket\") pod \"c9186dda-184a-47f6-913e-e3ba748da94d\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " Oct 30 05:56:39.158532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.158480 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-kube-api-access-5xtp2\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-kube-api-access-5xtp2\") pod \"c9186dda-184a-47f6-913e-e3ba748da94d\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " Oct 30 05:56:39.158532 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.158520 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-credential-socket\") pod \"c9186dda-184a-47f6-913e-e3ba748da94d\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " Oct 30 05:56:39.158851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.158724 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9186dda-184a-47f6-913e-e3ba748da94d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9186dda-184a-47f6-913e-e3ba748da94d-istiod-ca-cert\") pod \"c9186dda-184a-47f6-913e-e3ba748da94d\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " Oct 30 05:56:39.158851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.158770 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-istio-token\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-istio-token\") pod \"c9186dda-184a-47f6-913e-e3ba748da94d\" (UID: \"c9186dda-184a-47f6-913e-e3ba748da94d\") " Oct 30 05:56:39.159392 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.159357 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-socket" pod "c9186dda-184a-47f6-913e-e3ba748da94d" (UID: "c9186dda-184a-47f6-913e-e3ba748da94d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.160344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.159598 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-credential-socket" pod "c9186dda-184a-47f6-913e-e3ba748da94d" (UID: "c9186dda-184a-47f6-913e-e3ba748da94d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.160344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.159803 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-certs" pod "c9186dda-184a-47f6-913e-e3ba748da94d" (UID: "c9186dda-184a-47f6-913e-e3ba748da94d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.160344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.160123 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9186dda-184a-47f6-913e-e3ba748da94d-istiod-ca-cert" pod "c9186dda-184a-47f6-913e-e3ba748da94d" (UID: "c9186dda-184a-47f6-913e-e3ba748da94d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:39.161440 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.161412 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-data" pod "c9186dda-184a-47f6-913e-e3ba748da94d" (UID: "c9186dda-184a-47f6-913e-e3ba748da94d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.164298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.164275 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.164421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.164303 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.164421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.164317 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.164421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.164330 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.164421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.164342 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9186dda-184a-47f6-913e-e3ba748da94d-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.170928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.169420 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl"] Oct 30 05:56:39.170928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.170835 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-envoy" pod "c9186dda-184a-47f6-913e-e3ba748da94d" (UID: "c9186dda-184a-47f6-913e-e3ba748da94d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.171528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.171499 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-kube-api-access-5xtp2" pod "c9186dda-184a-47f6-913e-e3ba748da94d" (UID: "c9186dda-184a-47f6-913e-e3ba748da94d"). InnerVolumeSpecName "kube-api-access-5xtp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:39.172872 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.172833 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9186dda-184a-47f6-913e-e3ba748da94d-istio-podinfo" pod "c9186dda-184a-47f6-913e-e3ba748da94d" (UID: "c9186dda-184a-47f6-913e-e3ba748da94d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:56:39.173031 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.173010 5072 cpu_manager.go:414] "containerMap: removing container" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" Oct 30 05:56:39.173178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.173035 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" Oct 30 05:56:39.173335 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.173318 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" containerName="istio-proxy" Oct 30 05:56:39.177855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.177835 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.182998 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.182973 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-istio-token" pod "c9186dda-184a-47f6-913e-e3ba748da94d" (UID: "c9186dda-184a-47f6-913e-e3ba748da94d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:39.209446 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.209416 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl"] Oct 30 05:56:39.226547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.225043 5072 generic.go:356] "Generic (PLEG): container finished" podID="c9186dda-184a-47f6-913e-e3ba748da94d" containerID="ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686" exitCode=0 Oct 30 05:56:39.226547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.225235 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" Oct 30 05:56:39.226547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.226089 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" event={"ID":"c9186dda-184a-47f6-913e-e3ba748da94d","Type":"ContainerDied","Data":"ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686"} Oct 30 05:56:39.226547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.226142 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq" event={"ID":"c9186dda-184a-47f6-913e-e3ba748da94d","Type":"ContainerDied","Data":"dcb8cdc55ca845aabb2511371efb23b805aee59b5adb46e6826dc4fa42c5384b"} Oct 30 05:56:39.226547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.226172 5072 scope.go:122] "RemoveContainer" containerID="ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686" Oct 30 05:56:39.240722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.239304 5072 generic.go:356] "Generic (PLEG): container finished" podID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerID="83c6a223383342c8172709a641a524087332d6c65a97330203d813281fe10014" exitCode=0 Oct 30 05:56:39.240722 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.240333 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" event={"ID":"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a","Type":"ContainerDied","Data":"83c6a223383342c8172709a641a524087332d6c65a97330203d813281fe10014"} Oct 30 05:56:39.258557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.258536 5072 scope.go:122] "RemoveContainer" containerID="ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686" Oct 30 05:56:39.259002 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:56:39.258979 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686\": container with ID starting with ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686 not found: ID does not exist" containerID="ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686" Oct 30 05:56:39.259087 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.259013 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686"} err="failed to get container status \"ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686\": rpc error: code = NotFound desc = could not find container \"ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686\": container with ID starting with ba2b1b5c44a624f5d620f12e7966aa6529dad2d297ea83a6a96a81964a794686 not found: ID does not exist" Oct 30 05:56:39.265702 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.265665 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9hjq\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-kube-api-access-r9hjq\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.265809 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.265729 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.265934 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.265915 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.266034 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266017 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.266143 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266052 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.266203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266169 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.266203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266195 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.266316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266247 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96211f7-edc8-4516-a3be-9b9f22c2d995-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.266421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266403 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.266669 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266585 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9186dda-184a-47f6-913e-e3ba748da94d-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.266669 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266609 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtp2\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-kube-api-access-5xtp2\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.266669 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266624 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9186dda-184a-47f6-913e-e3ba748da94d-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.266669 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.266638 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9186dda-184a-47f6-913e-e3ba748da94d-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.316621 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.316583 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq"] Oct 30 05:56:39.405222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.405183 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.415037 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.414999 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hjq\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-kube-api-access-r9hjq\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.415340 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.415315 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.415649 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.415631 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.425574 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.425437 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-credential-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.428201 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.428148 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.428451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.428432 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.428824 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.428800 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.429030 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.428995 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.429239 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.429221 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96211f7-edc8-4516-a3be-9b9f22c2d995-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.431047 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.431012 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96211f7-edc8-4516-a3be-9b9f22c2d995-istiod-ca-cert\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.431619 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.431083 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-p4xhq"] Oct 30 05:56:39.432975 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.432938 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-socket\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.434342 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.434295 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-envoy\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.434762 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.434744 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-certs\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.434876 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.434805 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-podinfo\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.435136 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.435116 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-data\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.438025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.438008 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hjq\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-kube-api-access-r9hjq\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.456164 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.456137 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-token\") pod \"e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.474905 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.474875 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:56:39.541773 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.541739 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm"] Oct 30 05:56:39.542788 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.542755 5072 cpu_manager.go:414] "containerMap: removing container" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" Oct 30 05:56:39.542788 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.542778 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" Oct 30 05:56:39.542981 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.542967 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" containerName="istio-proxy" Oct 30 05:56:39.547071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.547052 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.560810 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.560781 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-39147c-openshift-default-dockercfg-p8mp2\"" Oct 30 05:56:39.571178 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.571153 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:39.571688 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.571662 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm"] Oct 30 05:56:39.632819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.632730 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istiod-ca-cert\") pod \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.633094 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-credential-socket\") pod \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.633150 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-certs\") pod \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.633221 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-kube-api-access-8lmfr\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-kube-api-access-8lmfr\") pod \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.633353 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-envoy\") pod \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.633425 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-socket\") pod \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.633572 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-podinfo\") pod \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.633631 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-token\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-token\") pod \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.633735 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-data\") pod \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\" (UID: \"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a\") " Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.634127 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-data\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.634291 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istiod-ca-cert\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.634322 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-socket\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.634414 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-credential-socket\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.634443 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-envoy\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.634568 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5cb7\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-kube-api-access-r5cb7\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.634603 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-podinfo\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.634702 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-token\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.634841 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-certs\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.635193 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-data" pod "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" (UID: "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.635747 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.635552 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istiod-ca-cert" pod "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" (UID: "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:39.636879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.635806 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-socket" pod "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" (UID: "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.636879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.636367 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-credential-socket" pod "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" (UID: "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.636879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.636615 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-certs" pod "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" (UID: "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.657135 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.657040 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-podinfo" pod "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" (UID: "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:56:39.676269 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.675444 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-envoy" pod "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" (UID: "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:39.676269 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.675716 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-token" pod "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" (UID: "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:39.677819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.677794 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-kube-api-access-8lmfr" pod "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" (UID: "c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a"). InnerVolumeSpecName "kube-api-access-8lmfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:39.736337 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.736271 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istiod-ca-cert\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.738137 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738073 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istiod-ca-cert\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.738137 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738123 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-socket\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.738334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738218 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-credential-socket\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.738334 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738243 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-envoy\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.738439 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738331 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5cb7\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-kube-api-access-r5cb7\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.738439 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738367 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-podinfo\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.738630 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738496 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-token\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.738800 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738628 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-certs\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.738800 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738765 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-data\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738978 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.738998 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739014 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739026 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmfr\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-kube-api-access-8lmfr\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739038 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739050 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739061 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739074 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739088 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739388 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-data\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739432 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-socket\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.739837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.739671 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-credential-socket\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.740541 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.740277 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-certs\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.743799 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.743751 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-podinfo\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.744144 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.744118 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-envoy\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.760767 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.760740 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5cb7\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-kube-api-access-r5cb7\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.761888 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.761868 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-token\") pod \"e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.867421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.867363 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:39.935814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.935636 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl"] Oct 30 05:56:39.982245 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.982196 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:39.982585 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.982566 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:39.982761 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:39.982744 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:40.183395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.183366 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm"] Oct 30 05:56:40.188793 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.188757 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:40.188894 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.188827 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:40.188894 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.188860 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:56:40.261509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.259191 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" Oct 30 05:56:40.261509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.259213 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6" event={"ID":"c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a","Type":"ContainerDied","Data":"fddddcec816c4665885afb56dc78c14335d659f17d4130bf275991e5e3734e28"} Oct 30 05:56:40.261509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.259294 5072 scope.go:122] "RemoveContainer" containerID="83c6a223383342c8172709a641a524087332d6c65a97330203d813281fe10014" Oct 30 05:56:40.264235 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.264205 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" event={"ID":"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9","Type":"ContainerStarted","Data":"88313720736d54ba6b9da1947f5f274a379ad408bee6c47053b18acfdfe0a298"} Oct 30 05:56:40.267027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.266998 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" event={"ID":"b96211f7-edc8-4516-a3be-9b9f22c2d995","Type":"ContainerStarted","Data":"ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c"} Oct 30 05:56:40.267142 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.267046 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" event={"ID":"b96211f7-edc8-4516-a3be-9b9f22c2d995","Type":"ContainerStarted","Data":"9fda0fda37bf13bdadf3a05e8edbd051f3ea6bbe57af1f13cd07c6310e927e5f"} Oct 30 05:56:40.319932 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.319870 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" podStartSLOduration=19.319851541 podStartE2EDuration="19.319851541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:40.318509564 +0000 UTC m=+1559.922198849" watchObservedRunningTime="2026-10-30 05:56:40.319851541 +0000 UTC m=+1559.923540799" Oct 30 05:56:40.362022 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.361989 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6"] Oct 30 05:56:40.366485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.366443 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-9nhd6"] Oct 30 05:56:40.525952 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.525875 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a" path="/var/lib/kubelet/pods/c1e7eac8-a2ee-4158-88d6-d4d8c4cb1c1a/volumes" Oct 30 05:56:40.526728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.526707 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9186dda-184a-47f6-913e-e3ba748da94d" path="/var/lib/kubelet/pods/c9186dda-184a-47f6-913e-e3ba748da94d/volumes" Oct 30 05:56:40.575480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:40.572014 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:41.279728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:41.279686 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" event={"ID":"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9","Type":"ContainerStarted","Data":"04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006"} Oct 30 05:56:41.349743 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:41.349688 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" podStartSLOduration=5.349675386 podStartE2EDuration="5.349675386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:41.335945503 +0000 UTC m=+1560.939634773" watchObservedRunningTime="2026-10-30 05:56:41.349675386 +0000 UTC m=+1560.953364642" Oct 30 05:56:41.581886 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:41.581798 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:56:41.582151 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:41.582122 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:56:41.874095 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:41.874054 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:41.898237 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:41.898204 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:42.294941 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:42.294851 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:42.296019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:42.295997 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:56:42.572908 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:42.572828 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:56:42.572908 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:42.572883 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:56:43.572580 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:43.572546 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:56:43.572981 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:43.572605 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:56:44.572279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:44.572250 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:56:44.572491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:44.572298 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:56:45.572145 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:45.572115 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:56:45.572551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:45.572183 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:56:45.647496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:45.643664 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd"] Oct 30 05:56:45.657943 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:45.651821 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:56:45.672489 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:45.670603 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-39147c\"" Oct 30 05:56:45.715487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:45.688042 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c94c6b-a6cf-4737-87c3-909cba482a5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd\" (UID: \"20c94c6b-a6cf-4737-87c3-909cba482a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:56:45.715487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:45.688398 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpsv\" (UniqueName: \"kubernetes.io/projected/20c94c6b-a6cf-4737-87c3-909cba482a5a-kube-api-access-dkpsv\") pod \"payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd\" (UID: \"20c94c6b-a6cf-4737-87c3-909cba482a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:56:46.081482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:45.981700 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-39147c-dockercfg-l8xxz\"" Oct 30 05:56:46.487529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:46.487431 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpsv\" (UniqueName: \"kubernetes.io/projected/20c94c6b-a6cf-4737-87c3-909cba482a5a-kube-api-access-dkpsv\") pod \"payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd\" (UID: \"20c94c6b-a6cf-4737-87c3-909cba482a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:56:46.487989 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:46.487965 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c94c6b-a6cf-4737-87c3-909cba482a5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd\" (UID: \"20c94c6b-a6cf-4737-87c3-909cba482a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:56:46.489430 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:46.489407 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c94c6b-a6cf-4737-87c3-909cba482a5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd\" (UID: \"20c94c6b-a6cf-4737-87c3-909cba482a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:56:46.494567 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:46.494525 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd"] Oct 30 05:56:46.542485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:46.541778 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpsv\" (UniqueName: \"kubernetes.io/projected/20c94c6b-a6cf-4737-87c3-909cba482a5a-kube-api-access-dkpsv\") pod \"payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd\" (UID: \"20c94c6b-a6cf-4737-87c3-909cba482a5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:56:46.572837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:46.572802 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:56:46.573230 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:46.572888 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 05:56:46.794027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:46.793927 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:56:47.119961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:47.119923 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd"] Oct 30 05:56:47.121296 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:56:47.121254 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c94c6b_a6cf_4737_87c3_909cba482a5a.slice/crio-17640f11d9a9f3e8250f35cb464caea7dce8590d78df1e694cbc867c63bc68da WatchSource:0}: Error finding container 17640f11d9a9f3e8250f35cb464caea7dce8590d78df1e694cbc867c63bc68da: Status 404 returned error can't find the container with id 17640f11d9a9f3e8250f35cb464caea7dce8590d78df1e694cbc867c63bc68da Oct 30 05:56:47.515254 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:47.515155 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" event={"ID":"20c94c6b-a6cf-4737-87c3-909cba482a5a","Type":"ContainerStarted","Data":"ce74df911b4a8f470e21b1768b83b4d385560c87e279bb14273b2d6051e960ff"} Oct 30 05:56:47.515254 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:47.515213 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" event={"ID":"20c94c6b-a6cf-4737-87c3-909cba482a5a","Type":"ContainerStarted","Data":"17640f11d9a9f3e8250f35cb464caea7dce8590d78df1e694cbc867c63bc68da"} Oct 30 05:56:47.515527 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:47.515432 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:56:47.557046 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:47.556980 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" podStartSLOduration=2.5569455960000003 podStartE2EDuration="2.556945596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:47.554586618 +0000 UTC m=+1567.158275877" watchObservedRunningTime="2026-10-30 05:56:47.556945596 +0000 UTC m=+1567.160634889" Oct 30 05:56:47.577003 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:47.576976 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:47.577444 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:47.577424 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:47.578279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:47.578259 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:56:47.717438 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:47.713010 5072 scope.go:122] "RemoveContainer" containerID="cc62c0a1387d6661e48e98e65068c998b46414481054fee719ce6a05051a94dd" Oct 30 05:56:58.548625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:56:58.548539 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 05:57:00.747488 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:57:00.746546 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:57:07.716635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:07.716588 5072 patch_prober.go:29] interesting pod/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:57:07.717185 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:07.716666 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:57:11.703886 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.703813 5072 generic.go:356] "Generic (PLEG): container finished" podID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerID="b21b202a79fcb11e93b95f25d6f654390b31e35ee448e4cf717622ecb141b845" exitCode=0 Oct 30 05:57:11.704327 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.703885 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" event={"ID":"8ee624f7-b2c8-48e1-b001-f2d58bc7a905","Type":"ContainerDied","Data":"b21b202a79fcb11e93b95f25d6f654390b31e35ee448e4cf717622ecb141b845"} Oct 30 05:57:11.704395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.704382 5072 scope.go:122] "RemoveContainer" containerID="b21b202a79fcb11e93b95f25d6f654390b31e35ee448e4cf717622ecb141b845" Oct 30 05:57:11.705811 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.705788 5072 generic.go:356] "Generic (PLEG): container finished" podID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerID="43193e033ef400484b2378d8dad6253081ff741d413ec465a67e7e765d3b6806" exitCode=0 Oct 30 05:57:11.705932 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.705839 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" event={"ID":"46689523-9454-44e3-8e76-fb8992bb6c1b","Type":"ContainerDied","Data":"43193e033ef400484b2378d8dad6253081ff741d413ec465a67e7e765d3b6806"} Oct 30 05:57:11.706208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.706191 5072 scope.go:122] "RemoveContainer" containerID="43193e033ef400484b2378d8dad6253081ff741d413ec465a67e7e765d3b6806" Oct 30 05:57:11.706528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.706504 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:57:11.706599 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.706586 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:57:11.706645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.706623 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:57:11.708052 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.708025 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:57:11.708161 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.708092 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:57:11.708161 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:11.708129 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:57:12.723441 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:12.723378 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" event={"ID":"8ee624f7-b2c8-48e1-b001-f2d58bc7a905","Type":"ContainerStarted","Data":"321bde7f2a923dcf50795b1a6b91ef7c0e0cc5720d48fdca0a3c357cbff3d1e8"} Oct 30 05:57:12.729493 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:12.728091 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" event={"ID":"46689523-9454-44e3-8e76-fb8992bb6c1b","Type":"ContainerStarted","Data":"cd8932f0f5a08da513ca3ced9e939a19b5cd80ddd027de2e178865a8ed60cc86"} Oct 30 05:57:13.473831 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:13.471365 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:57:13.473831 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:13.471573 5072 patch_prober.go:29] interesting pod/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:57:13.473831 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:13.471627 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 05:57:13.504199 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:13.503128 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:57:13.504199 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:13.503860 5072 patch_prober.go:29] interesting pod/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:57:13.504199 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:13.503913 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 05:57:14.478487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:14.476412 5072 patch_prober.go:29] interesting pod/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:57:14.478487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:14.476496 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 05:57:14.501792 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:14.501748 5072 patch_prober.go:29] interesting pod/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:57:14.501985 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:14.501820 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 05:57:15.470492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:15.469947 5072 patch_prober.go:29] interesting pod/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:57:15.470492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:15.470030 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 05:57:15.592297 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:15.591232 5072 patch_prober.go:29] interesting pod/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:57:15.592297 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:15.591299 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 05:57:16.489611 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:16.489577 5072 patch_prober.go:29] interesting pod/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:57:16.489898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:16.489873 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 05:57:16.502004 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:16.501980 5072 patch_prober.go:29] interesting pod/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:57:16.502202 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:16.502182 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 05:57:17.472068 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:17.472032 5072 patch_prober.go:29] interesting pod/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:57:17.472692 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:17.472659 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 05:57:17.501422 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:17.501397 5072 patch_prober.go:29] interesting pod/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:57:17.501619 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:17.501543 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 05:57:18.472914 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:18.472870 5072 patch_prober.go:29] interesting pod/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:57:18.478608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:18.475593 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 05:57:18.501925 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:18.501900 5072 patch_prober.go:29] interesting pod/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:57:18.502152 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:18.502105 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 05:57:18.968487 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:18.966018 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:57:19.469908 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:19.469861 5072 patch_prober.go:29] interesting pod/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:57:19.470130 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:19.469929 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 05:57:19.588338 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:19.588288 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:57:19.589954 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:19.589923 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:57:19.591000 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:19.590969 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:57:20.493485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:20.492770 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:57:20.501486 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:20.500414 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:57:30.713075 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:57:30.713030 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:57:47.994261 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:57:47.994221 5072 scope.go:122] "RemoveContainer" containerID="c8cc167164c721b991effc2f03897ae2be0f0c4cf72abe2e509441c48ee49be4" Oct 30 05:58:00.713324 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:00.713282 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:58:18.302627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:18.302571 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984"] Oct 30 05:58:18.303703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:18.303411 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" containerID="cri-o://00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e" gracePeriod=30 Oct 30 05:58:18.892250 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:18.891937 5072 patch_prober.go:29] interesting pod/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:58:18.892250 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:18.891992 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:58:23.052910 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.052869 5072 generic.go:356] "Generic (PLEG): container finished" podID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerID="04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006" exitCode=0 Oct 30 05:58:23.053328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.053181 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" event={"ID":"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9","Type":"ContainerDied","Data":"04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006"} Oct 30 05:58:23.053839 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.053816 5072 scope.go:122] "RemoveContainer" containerID="04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006" Oct 30 05:58:23.056548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.056514 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:58:23.056638 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.056619 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:58:23.056682 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.056671 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:58:23.823421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.823329 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:58:23.961535 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.954864 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z"] Oct 30 05:58:23.962041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.961987 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" podUID="e504e452-c65d-4e2a-8729-505c764d7f6b" containerName="maas-api" containerID="cri-o://6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338" gracePeriod=30 Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.990688 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-podinfo\") pod \"16e8eee0-9ec5-401b-9274-9a6319249b65\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.990738 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-token\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-token\") pod \"16e8eee0-9ec5-401b-9274-9a6319249b65\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.990801 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-envoy\") pod \"16e8eee0-9ec5-401b-9274-9a6319249b65\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.990840 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-kube-api-access-w98r2\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-kube-api-access-w98r2\") pod \"16e8eee0-9ec5-401b-9274-9a6319249b65\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.990936 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e8eee0-9ec5-401b-9274-9a6319249b65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e8eee0-9ec5-401b-9274-9a6319249b65-istiod-ca-cert\") pod \"16e8eee0-9ec5-401b-9274-9a6319249b65\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.990970 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-data\") pod \"16e8eee0-9ec5-401b-9274-9a6319249b65\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.991008 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-socket\") pod \"16e8eee0-9ec5-401b-9274-9a6319249b65\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.991091 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-certs\") pod \"16e8eee0-9ec5-401b-9274-9a6319249b65\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.991174 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-credential-socket\") pod \"16e8eee0-9ec5-401b-9274-9a6319249b65\" (UID: \"16e8eee0-9ec5-401b-9274-9a6319249b65\") " Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.991447 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-data" pod "16e8eee0-9ec5-401b-9274-9a6319249b65" (UID: "16e8eee0-9ec5-401b-9274-9a6319249b65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.991876 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-certs" pod "16e8eee0-9ec5-401b-9274-9a6319249b65" (UID: "16e8eee0-9ec5-401b-9274-9a6319249b65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.992047 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-credential-socket" pod "16e8eee0-9ec5-401b-9274-9a6319249b65" (UID: "16e8eee0-9ec5-401b-9274-9a6319249b65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.992189 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.992210 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.992231 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.992211 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-socket" pod "16e8eee0-9ec5-401b-9274-9a6319249b65" (UID: "16e8eee0-9ec5-401b-9274-9a6319249b65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.992525 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e8eee0-9ec5-401b-9274-9a6319249b65-istiod-ca-cert" pod "16e8eee0-9ec5-401b-9274-9a6319249b65" (UID: "16e8eee0-9ec5-401b-9274-9a6319249b65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:23.996516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.994513 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-podinfo" pod "16e8eee0-9ec5-401b-9274-9a6319249b65" (UID: "16e8eee0-9ec5-401b-9274-9a6319249b65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:24.002553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:23.999997 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-envoy" pod "16e8eee0-9ec5-401b-9274-9a6319249b65" (UID: "16e8eee0-9ec5-401b-9274-9a6319249b65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:24.002553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.000029 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-token" pod "16e8eee0-9ec5-401b-9274-9a6319249b65" (UID: "16e8eee0-9ec5-401b-9274-9a6319249b65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:24.002553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.001433 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-kube-api-access-w98r2" pod "16e8eee0-9ec5-401b-9274-9a6319249b65" (UID: "16e8eee0-9ec5-401b-9274-9a6319249b65"). InnerVolumeSpecName "kube-api-access-w98r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:24.066056 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.066023 5072 generic.go:356] "Generic (PLEG): container finished" podID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerID="00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e" exitCode=0 Oct 30 05:58:24.066575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.066121 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" event={"ID":"16e8eee0-9ec5-401b-9274-9a6319249b65","Type":"ContainerDied","Data":"00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e"} Oct 30 05:58:24.066575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.066176 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" event={"ID":"16e8eee0-9ec5-401b-9274-9a6319249b65","Type":"ContainerDied","Data":"0813c4fb98798c179504021bf40693fb97eb9c37dd6928bd8a648612362ef99f"} Oct 30 05:58:24.066575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.066212 5072 scope.go:122] "RemoveContainer" containerID="00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e" Oct 30 05:58:24.066575 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.066393 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984" Oct 30 05:58:24.072891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.071893 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" event={"ID":"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9","Type":"ContainerStarted","Data":"9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd"} Oct 30 05:58:24.090438 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.090415 5072 scope.go:122] "RemoveContainer" containerID="6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0" Oct 30 05:58:24.094207 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.094182 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:24.094311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.094214 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w98r2\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-kube-api-access-w98r2\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:24.094311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.094232 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e8eee0-9ec5-401b-9274-9a6319249b65-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:24.094311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.094248 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e8eee0-9ec5-401b-9274-9a6319249b65-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:24.094311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.094262 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:24.094311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.094275 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e8eee0-9ec5-401b-9274-9a6319249b65-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:24.102294 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.102269 5072 scope.go:122] "RemoveContainer" containerID="00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e" Oct 30 05:58:24.102671 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:24.102652 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e\": container with ID starting with 00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e not found: ID does not exist" containerID="00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e" Oct 30 05:58:24.102756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.102682 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e"} err="failed to get container status \"00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e\": rpc error: code = NotFound desc = could not find container \"00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e\": container with ID starting with 00418cd496ebb10afb538c769fcd020b957bb3195dcc198aad8733aa0010b41e not found: ID does not exist" Oct 30 05:58:24.102756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.102700 5072 scope.go:122] "RemoveContainer" containerID="6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0" Oct 30 05:58:24.103023 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:24.102987 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0\": container with ID starting with 6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0 not found: ID does not exist" containerID="6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0" Oct 30 05:58:24.103061 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.103029 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0"} err="failed to get container status \"6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0\": rpc error: code = NotFound desc = could not find container \"6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0\": container with ID starting with 6f784a43223aa4e91264556b139f2db5e00dbac7e8b32da7d648dde5c90eccc0 not found: ID does not exist" Oct 30 05:58:24.158010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.157974 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984"] Oct 30 05:58:24.158244 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.158223 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fd1c5d-openshift-default-dc764dd55-6h984"] Oct 30 05:58:24.893643 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.893609 5072 patch_prober.go:29] interesting pod/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:58:24.893890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:24.893863 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 05:58:25.017594 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.017131 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" path="/var/lib/kubelet/pods/16e8eee0-9ec5-401b-9274-9a6319249b65/volumes" Oct 30 05:58:25.021483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.017790 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:58:25.111482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.106824 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:58:25.130488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.116662 5072 generic.go:356] "Generic (PLEG): container finished" podID="e504e452-c65d-4e2a-8729-505c764d7f6b" containerID="6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338" exitCode=0 Oct 30 05:58:25.130488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.118157 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" Oct 30 05:58:25.130488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.118327 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" event={"ID":"e504e452-c65d-4e2a-8729-505c764d7f6b","Type":"ContainerDied","Data":"6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338"} Oct 30 05:58:25.130488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.118370 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z" event={"ID":"e504e452-c65d-4e2a-8729-505c764d7f6b","Type":"ContainerDied","Data":"5dfe65f7d113c32a90dfff78e4c33ce3fe1bf8e5f858870c1242e841d65ef28d"} Oct 30 05:58:25.130488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.118400 5072 scope.go:122] "RemoveContainer" containerID="6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338" Oct 30 05:58:25.148547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.147665 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-metrics-certs\") pod \"e504e452-c65d-4e2a-8729-505c764d7f6b\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " Oct 30 05:58:25.148547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.147836 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e504e452-c65d-4e2a-8729-505c764d7f6b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e504e452-c65d-4e2a-8729-505c764d7f6b-openshift-service-ca\") pod \"e504e452-c65d-4e2a-8729-505c764d7f6b\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " Oct 30 05:58:25.148547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.148028 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e504e452-c65d-4e2a-8729-505c764d7f6b-kube-api-access-wpp4p\" (UniqueName: \"kubernetes.io/projected/e504e452-c65d-4e2a-8729-505c764d7f6b-kube-api-access-wpp4p\") pod \"e504e452-c65d-4e2a-8729-505c764d7f6b\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " Oct 30 05:58:25.152756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.152704 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls\") pod \"e504e452-c65d-4e2a-8729-505c764d7f6b\" (UID: \"e504e452-c65d-4e2a-8729-505c764d7f6b\") " Oct 30 05:58:25.172985 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.170873 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e504e452-c65d-4e2a-8729-505c764d7f6b-openshift-service-ca" pod "e504e452-c65d-4e2a-8729-505c764d7f6b" (UID: "e504e452-c65d-4e2a-8729-505c764d7f6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:25.187750 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.187716 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e504e452-c65d-4e2a-8729-505c764d7f6b-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:25.198726 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.198691 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls" pod "e504e452-c65d-4e2a-8729-505c764d7f6b" (UID: "e504e452-c65d-4e2a-8729-505c764d7f6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:25.200006 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.199979 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e504e452-c65d-4e2a-8729-505c764d7f6b-kube-api-access-wpp4p" pod "e504e452-c65d-4e2a-8729-505c764d7f6b" (UID: "e504e452-c65d-4e2a-8729-505c764d7f6b"). InnerVolumeSpecName "kube-api-access-wpp4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:25.200636 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.200606 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-metrics-certs" pod "e504e452-c65d-4e2a-8729-505c764d7f6b" (UID: "e504e452-c65d-4e2a-8729-505c764d7f6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:25.267025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.264315 5072 scope.go:122] "RemoveContainer" containerID="6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338" Oct 30 05:58:25.267025 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:25.265603 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338\": container with ID starting with 6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338 not found: ID does not exist" containerID="6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338" Oct 30 05:58:25.267025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.265642 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338"} err="failed to get container status \"6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338\": rpc error: code = NotFound desc = could not find container \"6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338\": container with ID starting with 6b98aa6a628bc19c563457862a750067b557f714343e211f23713c8e2ae22338 not found: ID does not exist" Oct 30 05:58:25.289436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.289407 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:25.289436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.289432 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e504e452-c65d-4e2a-8729-505c764d7f6b-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:25.289436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.289442 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpp4p\" (UniqueName: \"kubernetes.io/projected/e504e452-c65d-4e2a-8729-505c764d7f6b-kube-api-access-wpp4p\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:25.497614 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.497579 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z"] Oct 30 05:58:25.501714 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.501689 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e94bd8-5d8bbcd57b-ml67z"] Oct 30 05:58:25.868279 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.868243 5072 patch_prober.go:29] interesting pod/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:58:25.868495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:25.868312 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 05:58:26.551063 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:26.551029 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e504e452-c65d-4e2a-8729-505c764d7f6b" path="/var/lib/kubelet/pods/e504e452-c65d-4e2a-8729-505c764d7f6b/volumes" Oct 30 05:58:26.867866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:26.867833 5072 patch_prober.go:29] interesting pod/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:58:26.868024 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:26.867886 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 05:58:27.294922 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:27.294821 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:58:27.875540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:27.873838 5072 patch_prober.go:29] interesting pod/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:58:27.875540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:27.873901 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 05:58:28.868742 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:28.868705 5072 patch_prober.go:29] interesting pod/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:58:28.868938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:28.868774 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 05:58:29.721490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:29.706707 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d"] Oct 30 05:58:29.721490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:29.706993 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" containerID="cri-o://bed074e8db8c6a68a16c2a085ad931896e80d7d72b9b1d776ab9363f66975cf3" gracePeriod=30 Oct 30 05:58:29.958104 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:29.925736 5072 patch_prober.go:29] interesting pod/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:58:29.959169 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:29.959132 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 05:58:30.085413 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.085236 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n"] Oct 30 05:58:30.137858 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.137815 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" containerID="cri-o://0a3da5637fe8cfe81d82b6c02cdd66e607e3941255d15e6bbd1fc191e3efe9e5" gracePeriod=30 Oct 30 05:58:30.287483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.278891 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk"] Oct 30 05:58:30.287483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.281234 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" containerID="cri-o://37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6" gracePeriod=30 Oct 30 05:58:30.371984 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.364640 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq"] Oct 30 05:58:30.371984 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.365499 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" containerID="cri-o://8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404" gracePeriod=30 Oct 30 05:58:30.453482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.453410 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f"] Oct 30 05:58:30.454686 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.453731 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" containerID="cri-o://e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13" gracePeriod=30 Oct 30 05:58:30.515064 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.515021 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-llr7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:58:30.515262 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.515094 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:58:30.564724 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.563970 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:58:30.564724 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.564203 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:58:30.607497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.607161 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-zz67d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:58:30.607497 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.607230 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:58:30.698802 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.698721 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8592n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:58:30.698802 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.698790 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:58:30.724534 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:30.720451 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:58:30.873806 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.873768 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:58:30.875175 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:30.875147 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 05:58:33.709105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.709071 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9"] Oct 30 05:58:33.709518 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.709258 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" containerID="cri-o://ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388" gracePeriod=30 Oct 30 05:58:33.718205 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.718176 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5"] Oct 30 05:58:33.718427 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.718406 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" containerID="cri-o://28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d" gracePeriod=30 Oct 30 05:58:33.723282 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.723257 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d"] Oct 30 05:58:33.723501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.723478 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" containerID="cri-o://f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4" gracePeriod=30 Oct 30 05:58:33.727746 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.727723 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl"] Oct 30 05:58:33.727884 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.727867 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" containerID="cri-o://ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c" gracePeriod=30 Oct 30 05:58:33.734893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.734861 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8"] Oct 30 05:58:33.735091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:33.735070 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" containerID="cri-o://249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b" gracePeriod=30 Oct 30 05:58:35.406227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.405716 5072 generic.go:356] "Generic (PLEG): container finished" podID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerID="bed074e8db8c6a68a16c2a085ad931896e80d7d72b9b1d776ab9363f66975cf3" exitCode=0 Oct 30 05:58:35.406227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.405883 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" event={"ID":"0f42b70a-73d5-4f6c-9b09-afe535fda7f5","Type":"ContainerDied","Data":"bed074e8db8c6a68a16c2a085ad931896e80d7d72b9b1d776ab9363f66975cf3"} Oct 30 05:58:35.411784 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.409282 5072 generic.go:356] "Generic (PLEG): container finished" podID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerID="0a3da5637fe8cfe81d82b6c02cdd66e607e3941255d15e6bbd1fc191e3efe9e5" exitCode=0 Oct 30 05:58:35.411784 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.409488 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" event={"ID":"02b09b1b-4429-4301-be01-fa1b1ad0670a","Type":"ContainerDied","Data":"0a3da5637fe8cfe81d82b6c02cdd66e607e3941255d15e6bbd1fc191e3efe9e5"} Oct 30 05:58:35.664547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.664107 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:58:35.741867 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.741837 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:58:35.744253 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.744218 5072 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 05:58:35.744370 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.744272 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.798857 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-token\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-token\") pod \"02b09b1b-4429-4301-be01-fa1b1ad0670a\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.798911 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-credential-socket\") pod \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.798949 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-podinfo\") pod \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.798988 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-certs\") pod \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799091 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istiod-ca-cert\") pod \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799164 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-data\") pod \"02b09b1b-4429-4301-be01-fa1b1ad0670a\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799214 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-socket\") pod \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799260 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-kube-api-access-wx7bp\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-kube-api-access-wx7bp\") pod \"02b09b1b-4429-4301-be01-fa1b1ad0670a\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799286 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-token\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-token\") pod \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799330 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-certs\") pod \"02b09b1b-4429-4301-be01-fa1b1ad0670a\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799380 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-envoy\") pod \"02b09b1b-4429-4301-be01-fa1b1ad0670a\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799407 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-data\") pod \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799478 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-envoy\") pod \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799517 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b09b1b-4429-4301-be01-fa1b1ad0670a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b09b1b-4429-4301-be01-fa1b1ad0670a-istiod-ca-cert\") pod \"02b09b1b-4429-4301-be01-fa1b1ad0670a\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799621 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-credential-socket\") pod \"02b09b1b-4429-4301-be01-fa1b1ad0670a\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799671 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-kube-api-access-cmvxn\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-kube-api-access-cmvxn\") pod \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\" (UID: \"0f42b70a-73d5-4f6c-9b09-afe535fda7f5\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799758 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-socket\") pod \"02b09b1b-4429-4301-be01-fa1b1ad0670a\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " Oct 30 05:58:35.805353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.799855 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-podinfo\") pod \"02b09b1b-4429-4301-be01-fa1b1ad0670a\" (UID: \"02b09b1b-4429-4301-be01-fa1b1ad0670a\") " Oct 30 05:58:35.824865 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.813045 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-credential-socket" pod "0f42b70a-73d5-4f6c-9b09-afe535fda7f5" (UID: "0f42b70a-73d5-4f6c-9b09-afe535fda7f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.824865 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.813389 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-certs" pod "02b09b1b-4429-4301-be01-fa1b1ad0670a" (UID: "02b09b1b-4429-4301-be01-fa1b1ad0670a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.824865 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.814182 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b09b1b-4429-4301-be01-fa1b1ad0670a-istiod-ca-cert" pod "02b09b1b-4429-4301-be01-fa1b1ad0670a" (UID: "02b09b1b-4429-4301-be01-fa1b1ad0670a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:35.829598 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.829555 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-credential-socket" pod "02b09b1b-4429-4301-be01-fa1b1ad0670a" (UID: "02b09b1b-4429-4301-be01-fa1b1ad0670a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.834540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.834439 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-socket" pod "02b09b1b-4429-4301-be01-fa1b1ad0670a" (UID: "02b09b1b-4429-4301-be01-fa1b1ad0670a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.850525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.835700 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-envoy" pod "02b09b1b-4429-4301-be01-fa1b1ad0670a" (UID: "02b09b1b-4429-4301-be01-fa1b1ad0670a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.850525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.836343 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-socket" pod "0f42b70a-73d5-4f6c-9b09-afe535fda7f5" (UID: "0f42b70a-73d5-4f6c-9b09-afe535fda7f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.850525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.836685 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istiod-ca-cert" pod "0f42b70a-73d5-4f6c-9b09-afe535fda7f5" (UID: "0f42b70a-73d5-4f6c-9b09-afe535fda7f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:35.850525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.836952 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-data" pod "02b09b1b-4429-4301-be01-fa1b1ad0670a" (UID: "02b09b1b-4429-4301-be01-fa1b1ad0670a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.850525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.839534 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-certs" pod "0f42b70a-73d5-4f6c-9b09-afe535fda7f5" (UID: "0f42b70a-73d5-4f6c-9b09-afe535fda7f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.850525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.840038 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-token" pod "0f42b70a-73d5-4f6c-9b09-afe535fda7f5" (UID: "0f42b70a-73d5-4f6c-9b09-afe535fda7f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:35.850525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.840555 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-envoy" pod "0f42b70a-73d5-4f6c-9b09-afe535fda7f5" (UID: "0f42b70a-73d5-4f6c-9b09-afe535fda7f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.850525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.844628 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-kube-api-access-wx7bp" pod "02b09b1b-4429-4301-be01-fa1b1ad0670a" (UID: "02b09b1b-4429-4301-be01-fa1b1ad0670a"). InnerVolumeSpecName "kube-api-access-wx7bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:35.893553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.892015 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-data" pod "0f42b70a-73d5-4f6c-9b09-afe535fda7f5" (UID: "0f42b70a-73d5-4f6c-9b09-afe535fda7f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.899953 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.898707 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-kube-api-access-cmvxn" pod "0f42b70a-73d5-4f6c-9b09-afe535fda7f5" (UID: "0f42b70a-73d5-4f6c-9b09-afe535fda7f5"). InnerVolumeSpecName "kube-api-access-cmvxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905581 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905611 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx7bp\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-kube-api-access-wx7bp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905626 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905640 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905655 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905668 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905680 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905697 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02b09b1b-4429-4301-be01-fa1b1ad0670a-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905709 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905723 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmvxn\" (UniqueName: \"kubernetes.io/projected/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-kube-api-access-cmvxn\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905741 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905755 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905768 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905784 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.905796 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.907298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.906513 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-podinfo" pod "0f42b70a-73d5-4f6c-9b09-afe535fda7f5" (UID: "0f42b70a-73d5-4f6c-9b09-afe535fda7f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:35.909670 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.909637 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-podinfo" pod "02b09b1b-4429-4301-be01-fa1b1ad0670a" (UID: "02b09b1b-4429-4301-be01-fa1b1ad0670a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:35.911341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.909943 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:58:35.913318 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:35.913264 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-token" pod "02b09b1b-4429-4301-be01-fa1b1ad0670a" (UID: "02b09b1b-4429-4301-be01-fa1b1ad0670a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:36.006479 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.006428 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-envoy\") pod \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " Oct 30 05:58:36.006668 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.006645 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-socket\") pod \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " Oct 30 05:58:36.006731 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.006695 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-kube-api-access-l5gr5\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-kube-api-access-l5gr5\") pod \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " Oct 30 05:58:36.006791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.006757 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-certs\") pod \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " Oct 30 05:58:36.006848 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.006811 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-credential-socket\") pod \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " Oct 30 05:58:36.006918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.006902 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istiod-ca-cert\") pod \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " Oct 30 05:58:36.007059 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.006928 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-data\") pod \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " Oct 30 05:58:36.007059 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.006986 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-token\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-token\") pod \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " Oct 30 05:58:36.007059 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.007033 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-podinfo\") pod \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\" (UID: \"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c\") " Oct 30 05:58:36.007688 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.007667 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.007688 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.007687 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02b09b1b-4429-4301-be01-fa1b1ad0670a-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.007866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.007696 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42b70a-73d5-4f6c-9b09-afe535fda7f5-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.009139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.009109 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-socket" pod "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" (UID: "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.009292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.009275 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-certs" pod "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" (UID: "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.009410 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.009395 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-data" pod "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" (UID: "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.009639 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.009622 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istiod-ca-cert" pod "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" (UID: "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:36.011769 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.011743 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-envoy" pod "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" (UID: "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.012129 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.012108 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-podinfo" pod "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" (UID: "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:36.012256 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.012233 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-credential-socket" pod "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" (UID: "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.012833 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.012811 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-token" pod "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" (UID: "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:36.014871 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.014844 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-kube-api-access-l5gr5" pod "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" (UID: "ee667dc4-ddc3-49c6-b87f-02bc14d31a6c"). InnerVolumeSpecName "kube-api-access-l5gr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:36.059222 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.059195 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:58:36.062534 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.062512 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:58:36.108410 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108378 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-socket\") pod \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " Oct 30 05:58:36.108857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108440 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-credential-socket\") pod \"5fd3999c-39f6-4edf-9b57-71492639891d\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " Oct 30 05:58:36.108857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108543 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-kube-api-access-mt8mj\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-kube-api-access-mt8mj\") pod \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " Oct 30 05:58:36.108857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108626 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-certs\") pod \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " Oct 30 05:58:36.108857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108660 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-envoy\") pod \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " Oct 30 05:58:36.108857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108746 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-kube-api-access-rh7ww\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-kube-api-access-rh7ww\") pod \"5fd3999c-39f6-4edf-9b57-71492639891d\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " Oct 30 05:58:36.108857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108776 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-socket\") pod \"5fd3999c-39f6-4edf-9b57-71492639891d\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " Oct 30 05:58:36.108857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108817 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-envoy\") pod \"5fd3999c-39f6-4edf-9b57-71492639891d\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " Oct 30 05:58:36.108857 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108847 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istiod-ca-cert\") pod \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108868 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-credential-socket" pod "5fd3999c-39f6-4edf-9b57-71492639891d" (UID: "5fd3999c-39f6-4edf-9b57-71492639891d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108876 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fd3999c-39f6-4edf-9b57-71492639891d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd3999c-39f6-4edf-9b57-71492639891d-istio-podinfo\") pod \"5fd3999c-39f6-4edf-9b57-71492639891d\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.108970 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-istio-token\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-istio-token\") pod \"5fd3999c-39f6-4edf-9b57-71492639891d\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.109028 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-token\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-token\") pod \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.109046 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-data\") pod \"5fd3999c-39f6-4edf-9b57-71492639891d\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.109102 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-credential-socket\") pod \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.109132 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-certs\") pod \"5fd3999c-39f6-4edf-9b57-71492639891d\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.109182 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-podinfo\") pod \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.109237 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-data\") pod \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\" (UID: \"dbeaf7e5-398d-4ef7-a2b1-310d653279a2\") " Oct 30 05:58:36.109355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.109281 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd3999c-39f6-4edf-9b57-71492639891d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd3999c-39f6-4edf-9b57-71492639891d-istiod-ca-cert\") pod \"5fd3999c-39f6-4edf-9b57-71492639891d\" (UID: \"5fd3999c-39f6-4edf-9b57-71492639891d\") " Oct 30 05:58:36.109922 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.109486 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-credential-socket" pod "dbeaf7e5-398d-4ef7-a2b1-310d653279a2" (UID: "dbeaf7e5-398d-4ef7-a2b1-310d653279a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.109922 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.109670 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-data" pod "5fd3999c-39f6-4edf-9b57-71492639891d" (UID: "5fd3999c-39f6-4edf-9b57-71492639891d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.111055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.110243 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-certs" pod "5fd3999c-39f6-4edf-9b57-71492639891d" (UID: "5fd3999c-39f6-4edf-9b57-71492639891d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.111055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.110541 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-certs" pod "dbeaf7e5-398d-4ef7-a2b1-310d653279a2" (UID: "dbeaf7e5-398d-4ef7-a2b1-310d653279a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.111055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.110606 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd3999c-39f6-4edf-9b57-71492639891d-istiod-ca-cert" pod "5fd3999c-39f6-4edf-9b57-71492639891d" (UID: "5fd3999c-39f6-4edf-9b57-71492639891d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:36.111055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.110997 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111020 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111030 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-socket" pod "5fd3999c-39f6-4edf-9b57-71492639891d" (UID: "5fd3999c-39f6-4edf-9b57-71492639891d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.111055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111034 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111061 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111087 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd3999c-39f6-4edf-9b57-71492639891d-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111101 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111111 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istiod-ca-cert" pod "dbeaf7e5-398d-4ef7-a2b1-310d653279a2" (UID: "dbeaf7e5-398d-4ef7-a2b1-310d653279a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111124 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5gr5\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-kube-api-access-l5gr5\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111139 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111153 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111164 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111174 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-data" pod "dbeaf7e5-398d-4ef7-a2b1-310d653279a2" (UID: "dbeaf7e5-398d-4ef7-a2b1-310d653279a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111178 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111209 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111223 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111238 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.111555 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.111250 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.112509 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.112449 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-token" pod "dbeaf7e5-398d-4ef7-a2b1-310d653279a2" (UID: "dbeaf7e5-398d-4ef7-a2b1-310d653279a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:36.113027 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.113003 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fd3999c-39f6-4edf-9b57-71492639891d-istio-podinfo" pod "5fd3999c-39f6-4edf-9b57-71492639891d" (UID: "5fd3999c-39f6-4edf-9b57-71492639891d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:36.114783 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.113985 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-podinfo" pod "dbeaf7e5-398d-4ef7-a2b1-310d653279a2" (UID: "dbeaf7e5-398d-4ef7-a2b1-310d653279a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:36.114783 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.114489 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-envoy" pod "5fd3999c-39f6-4edf-9b57-71492639891d" (UID: "5fd3999c-39f6-4edf-9b57-71492639891d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.114783 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.114298 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-envoy" pod "dbeaf7e5-398d-4ef7-a2b1-310d653279a2" (UID: "dbeaf7e5-398d-4ef7-a2b1-310d653279a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.115089 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.115066 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-kube-api-access-mt8mj" pod "dbeaf7e5-398d-4ef7-a2b1-310d653279a2" (UID: "dbeaf7e5-398d-4ef7-a2b1-310d653279a2"). InnerVolumeSpecName "kube-api-access-mt8mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:36.115311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.115294 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-socket" pod "dbeaf7e5-398d-4ef7-a2b1-310d653279a2" (UID: "dbeaf7e5-398d-4ef7-a2b1-310d653279a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:36.117301 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.117268 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-kube-api-access-rh7ww" pod "5fd3999c-39f6-4edf-9b57-71492639891d" (UID: "5fd3999c-39f6-4edf-9b57-71492639891d"). InnerVolumeSpecName "kube-api-access-rh7ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:36.117399 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.117329 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-istio-token" pod "5fd3999c-39f6-4edf-9b57-71492639891d" (UID: "5fd3999c-39f6-4edf-9b57-71492639891d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:36.213697 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213646 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213697 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213696 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213710 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213722 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213734 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213746 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt8mj\" (UniqueName: \"kubernetes.io/projected/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-kube-api-access-mt8mj\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213759 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213771 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh7ww\" (UniqueName: \"kubernetes.io/projected/5fd3999c-39f6-4edf-9b57-71492639891d-kube-api-access-rh7ww\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213783 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213795 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd3999c-39f6-4edf-9b57-71492639891d-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213806 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeaf7e5-398d-4ef7-a2b1-310d653279a2-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.213899 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.213817 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd3999c-39f6-4edf-9b57-71492639891d-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.419215 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.419181 5072 generic.go:356] "Generic (PLEG): container finished" podID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerID="e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13" exitCode=0 Oct 30 05:58:36.419750 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.419231 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" Oct 30 05:58:36.419750 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.419258 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" event={"ID":"dbeaf7e5-398d-4ef7-a2b1-310d653279a2","Type":"ContainerDied","Data":"e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13"} Oct 30 05:58:36.419750 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.419313 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f" event={"ID":"dbeaf7e5-398d-4ef7-a2b1-310d653279a2","Type":"ContainerDied","Data":"3bdb1d70a141948aaf213b02db73c34721061e3635a51cd498c8759a4e82c7b3"} Oct 30 05:58:36.419750 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.419345 5072 scope.go:122] "RemoveContainer" containerID="e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13" Oct 30 05:58:36.421492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.421182 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" event={"ID":"0f42b70a-73d5-4f6c-9b09-afe535fda7f5","Type":"ContainerDied","Data":"8f4839c654ebb2e1f5b3f437071b9f189af0fde6bffa8bb1af097f456862855e"} Oct 30 05:58:36.421492 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.421287 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d" Oct 30 05:58:36.422956 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.422935 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" Oct 30 05:58:36.423083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.423065 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n" event={"ID":"02b09b1b-4429-4301-be01-fa1b1ad0670a","Type":"ContainerDied","Data":"eb759b58d8d0aa2a14d3fe14b2328974f3e0fdb6f6536c555754f3757b0db751"} Oct 30 05:58:36.429391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.429364 5072 generic.go:356] "Generic (PLEG): container finished" podID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerID="8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404" exitCode=0 Oct 30 05:58:36.429613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.429593 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" event={"ID":"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c","Type":"ContainerDied","Data":"8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404"} Oct 30 05:58:36.429694 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.429641 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" event={"ID":"ee667dc4-ddc3-49c6-b87f-02bc14d31a6c","Type":"ContainerDied","Data":"d23b7df6de0212997d6d49d79cc28d8a0728011164b3a894ca80be93107ccf0f"} Oct 30 05:58:36.429765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.429711 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq" Oct 30 05:58:36.433851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.433805 5072 generic.go:356] "Generic (PLEG): container finished" podID="5fd3999c-39f6-4edf-9b57-71492639891d" containerID="37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6" exitCode=0 Oct 30 05:58:36.433851 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.433843 5072 scope.go:122] "RemoveContainer" containerID="e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13" Oct 30 05:58:36.434650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.433874 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" event={"ID":"5fd3999c-39f6-4edf-9b57-71492639891d","Type":"ContainerDied","Data":"37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6"} Oct 30 05:58:36.434650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.433914 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" event={"ID":"5fd3999c-39f6-4edf-9b57-71492639891d","Type":"ContainerDied","Data":"69ccd6c420df541d0b59408a84b3fbb94cc5740360500d7f6f5074823e391f83"} Oct 30 05:58:36.434650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.433952 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk" Oct 30 05:58:36.434650 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:36.434131 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13\": container with ID starting with e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13 not found: ID does not exist" containerID="e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13" Oct 30 05:58:36.434650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.434164 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13"} err="failed to get container status \"e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13\": rpc error: code = NotFound desc = could not find container \"e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13\": container with ID starting with e8b9650a1b3879d4827da9c2bd630028794a5fcb4b1142cb659e87f35e4e6f13 not found: ID does not exist" Oct 30 05:58:36.434650 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.434184 5072 scope.go:122] "RemoveContainer" containerID="bed074e8db8c6a68a16c2a085ad931896e80d7d72b9b1d776ab9363f66975cf3" Oct 30 05:58:36.448399 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.448378 5072 scope.go:122] "RemoveContainer" containerID="0a3da5637fe8cfe81d82b6c02cdd66e607e3941255d15e6bbd1fc191e3efe9e5" Oct 30 05:58:36.461777 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.461752 5072 scope.go:122] "RemoveContainer" containerID="8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404" Oct 30 05:58:36.471713 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.471689 5072 scope.go:122] "RemoveContainer" containerID="8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404" Oct 30 05:58:36.472020 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:36.472000 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404\": container with ID starting with 8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404 not found: ID does not exist" containerID="8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404" Oct 30 05:58:36.472093 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.472030 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404"} err="failed to get container status \"8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404\": rpc error: code = NotFound desc = could not find container \"8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404\": container with ID starting with 8c7834c2a84e6801419eb71b456d5c44fff72e9ac7e111414ce0a90984e3d404 not found: ID does not exist" Oct 30 05:58:36.472093 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.472049 5072 scope.go:122] "RemoveContainer" containerID="37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6" Oct 30 05:58:36.483501 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.483475 5072 scope.go:122] "RemoveContainer" containerID="611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f" Oct 30 05:58:36.495169 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.495148 5072 scope.go:122] "RemoveContainer" containerID="37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6" Oct 30 05:58:36.495535 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:36.495505 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6\": container with ID starting with 37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6 not found: ID does not exist" containerID="37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6" Oct 30 05:58:36.495670 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.495548 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6"} err="failed to get container status \"37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6\": rpc error: code = NotFound desc = could not find container \"37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6\": container with ID starting with 37b037b6edeadce3255fa9704312917b3b1b6c5fc6c4abfaebfb89e29e4656a6 not found: ID does not exist" Oct 30 05:58:36.495670 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.495576 5072 scope.go:122] "RemoveContainer" containerID="611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f" Oct 30 05:58:36.495969 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:36.495942 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f\": container with ID starting with 611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f not found: ID does not exist" containerID="611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f" Oct 30 05:58:36.496082 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.495977 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f"} err="failed to get container status \"611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f\": rpc error: code = NotFound desc = could not find container \"611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f\": container with ID starting with 611bdcd8cea40a421cd6f4462a8cc56caeee159fd0a2706637d412fbb0e2796f not found: ID does not exist" Oct 30 05:58:36.510502 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.510435 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d"] Oct 30 05:58:36.523529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.523500 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-zz67d"] Oct 30 05:58:36.551091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.551063 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq"] Oct 30 05:58:36.555017 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.554991 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-pzvfq"] Oct 30 05:58:36.583573 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.583542 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk"] Oct 30 05:58:36.587400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.587381 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8jgfk"] Oct 30 05:58:36.618853 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.618819 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f"] Oct 30 05:58:36.624833 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.624800 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-llr7f"] Oct 30 05:58:36.655205 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.655162 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n"] Oct 30 05:58:36.659216 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:36.659193 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8592n"] Oct 30 05:58:37.036952 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.036917 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t"] Oct 30 05:58:37.041078 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041050 5072 cpu_manager.go:414] "containerMap: removing container" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" Oct 30 05:58:37.041274 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041250 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" Oct 30 05:58:37.041405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041294 5072 cpu_manager.go:414] "containerMap: removing container" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" Oct 30 05:58:37.041405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041303 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" Oct 30 05:58:37.041405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041317 5072 cpu_manager.go:414] "containerMap: removing container" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" Oct 30 05:58:37.041405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041324 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" Oct 30 05:58:37.041405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041339 5072 cpu_manager.go:414] "containerMap: removing container" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" Oct 30 05:58:37.041405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041346 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" Oct 30 05:58:37.041405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041383 5072 cpu_manager.go:414] "containerMap: removing container" podUID="e504e452-c65d-4e2a-8729-505c764d7f6b" containerName="maas-api" Oct 30 05:58:37.041405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041391 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e504e452-c65d-4e2a-8729-505c764d7f6b" containerName="maas-api" Oct 30 05:58:37.041405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041406 5072 cpu_manager.go:414] "containerMap: removing container" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041414 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041450 5072 cpu_manager.go:414] "containerMap: removing container" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041473 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041494 5072 cpu_manager.go:414] "containerMap: removing container" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041501 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041692 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e504e452-c65d-4e2a-8729-505c764d7f6b" containerName="maas-api" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041703 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041715 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041726 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041734 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041743 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041755 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041766 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" containerName="istio-proxy" Oct 30 05:58:37.041893 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.041773 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e8eee0-9ec5-401b-9274-9a6319249b65" containerName="istio-proxy" Oct 30 05:58:37.047985 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.047961 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.063794 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.063764 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0382c2-openshift-default-dockercfg-d2vb8\"" Oct 30 05:58:37.065292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.065264 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t"] Oct 30 05:58:37.130318 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.130282 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-certs\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.130318 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.130330 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab0bf3a-d180-4e27-a171-33b97bec9497-istiod-ca-cert\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.130610 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.130435 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-token\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.130610 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.130512 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-envoy\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.130610 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.130543 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-podinfo\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.130610 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.130585 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-socket\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.130776 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.130631 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-data\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.130776 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.130693 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-credential-socket\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.130846 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.130782 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzz29\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-kube-api-access-pzz29\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.232273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232234 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzz29\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-kube-api-access-pzz29\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.232524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232363 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-certs\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.232524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232402 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab0bf3a-d180-4e27-a171-33b97bec9497-istiod-ca-cert\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.232524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232506 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-token\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.232765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232536 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-envoy\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.232765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232581 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-podinfo\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.232765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232648 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-socket\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.232765 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232732 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-data\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.233010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232834 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-certs\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.233010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.232846 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-credential-socket\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.233173 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.233144 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-credential-socket\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.233324 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.233299 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-data\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.233402 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.233326 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab0bf3a-d180-4e27-a171-33b97bec9497-istiod-ca-cert\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.233528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.233509 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-socket\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.235432 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.235405 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-envoy\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.236046 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.236020 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-podinfo\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.255358 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.255311 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzz29\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-kube-api-access-pzz29\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.256519 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.256497 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-token\") pod \"e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.386254 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.383835 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:37.899512 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.899419 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t"] Oct 30 05:58:37.903999 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:58:37.903966 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab0bf3a_d180_4e27_a171_33b97bec9497.slice/crio-99c93c1c792b21b2661ce3c26ac74913e012dc6ca0af3bface49b20b35a3c3ce WatchSource:0}: Error finding container 99c93c1c792b21b2661ce3c26ac74913e012dc6ca0af3bface49b20b35a3c3ce: Status 404 returned error can't find the container with id 99c93c1c792b21b2661ce3c26ac74913e012dc6ca0af3bface49b20b35a3c3ce Oct 30 05:58:37.907185 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.907155 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:58:37.907285 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.907239 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:58:37.907285 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:37.907275 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:58:38.075771 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.075735 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:58:38.075918 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.075802 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:58:38.101157 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.101111 5072 patch_prober.go:29] interesting pod/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:58:38.101331 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.101173 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:58:38.522128 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.522098 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b09b1b-4429-4301-be01-fa1b1ad0670a" path="/var/lib/kubelet/pods/02b09b1b-4429-4301-be01-fa1b1ad0670a/volumes" Oct 30 05:58:38.522751 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.522727 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f42b70a-73d5-4f6c-9b09-afe535fda7f5" path="/var/lib/kubelet/pods/0f42b70a-73d5-4f6c-9b09-afe535fda7f5/volumes" Oct 30 05:58:38.523197 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.523181 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" path="/var/lib/kubelet/pods/5fd3999c-39f6-4edf-9b57-71492639891d/volumes" Oct 30 05:58:38.524021 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.524000 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbeaf7e5-398d-4ef7-a2b1-310d653279a2" path="/var/lib/kubelet/pods/dbeaf7e5-398d-4ef7-a2b1-310d653279a2/volumes" Oct 30 05:58:38.524450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.524432 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee667dc4-ddc3-49c6-b87f-02bc14d31a6c" path="/var/lib/kubelet/pods/ee667dc4-ddc3-49c6-b87f-02bc14d31a6c/volumes" Oct 30 05:58:38.541607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.539610 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" event={"ID":"fab0bf3a-d180-4e27-a171-33b97bec9497","Type":"ContainerStarted","Data":"e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0"} Oct 30 05:58:38.541607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.539670 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" event={"ID":"fab0bf3a-d180-4e27-a171-33b97bec9497","Type":"ContainerStarted","Data":"99c93c1c792b21b2661ce3c26ac74913e012dc6ca0af3bface49b20b35a3c3ce"} Oct 30 05:58:38.584933 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:38.584868 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podStartSLOduration=2.584848645 podStartE2EDuration="2.584848645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:38.581885135 +0000 UTC m=+1678.185574409" watchObservedRunningTime="2026-10-30 05:58:38.584848645 +0000 UTC m=+1678.188537904" Oct 30 05:58:39.188671 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.188622 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.268628 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-certs\") pod \"b96211f7-edc8-4516-a3be-9b9f22c2d995\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.268690 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-credential-socket\") pod \"b96211f7-edc8-4516-a3be-9b9f22c2d995\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.268749 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-token\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-token\") pod \"b96211f7-edc8-4516-a3be-9b9f22c2d995\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.268900 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-certs" pod "b96211f7-edc8-4516-a3be-9b9f22c2d995" (UID: "b96211f7-edc8-4516-a3be-9b9f22c2d995"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.268912 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-podinfo\") pod \"b96211f7-edc8-4516-a3be-9b9f22c2d995\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.268958 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-envoy\") pod \"b96211f7-edc8-4516-a3be-9b9f22c2d995\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.268983 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-credential-socket" pod "b96211f7-edc8-4516-a3be-9b9f22c2d995" (UID: "b96211f7-edc8-4516-a3be-9b9f22c2d995"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.269025 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-kube-api-access-r9hjq\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-kube-api-access-r9hjq\") pod \"b96211f7-edc8-4516-a3be-9b9f22c2d995\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.269078 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-data\") pod \"b96211f7-edc8-4516-a3be-9b9f22c2d995\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.269106 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96211f7-edc8-4516-a3be-9b9f22c2d995-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96211f7-edc8-4516-a3be-9b9f22c2d995-istiod-ca-cert\") pod \"b96211f7-edc8-4516-a3be-9b9f22c2d995\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " Oct 30 05:58:39.269349 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.269164 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-socket\") pod \"b96211f7-edc8-4516-a3be-9b9f22c2d995\" (UID: \"b96211f7-edc8-4516-a3be-9b9f22c2d995\") " Oct 30 05:58:39.270033 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.269689 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-data" pod "b96211f7-edc8-4516-a3be-9b9f22c2d995" (UID: "b96211f7-edc8-4516-a3be-9b9f22c2d995"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.270391 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.270347 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96211f7-edc8-4516-a3be-9b9f22c2d995-istiod-ca-cert" pod "b96211f7-edc8-4516-a3be-9b9f22c2d995" (UID: "b96211f7-edc8-4516-a3be-9b9f22c2d995"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:39.271314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.271290 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.271412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.271324 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96211f7-edc8-4516-a3be-9b9f22c2d995-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.271412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.271340 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.271412 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.271384 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.272074 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.272039 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-socket" pod "b96211f7-edc8-4516-a3be-9b9f22c2d995" (UID: "b96211f7-edc8-4516-a3be-9b9f22c2d995"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.289365 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.289328 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-envoy" pod "b96211f7-edc8-4516-a3be-9b9f22c2d995" (UID: "b96211f7-edc8-4516-a3be-9b9f22c2d995"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.289517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.289334 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-podinfo" pod "b96211f7-edc8-4516-a3be-9b9f22c2d995" (UID: "b96211f7-edc8-4516-a3be-9b9f22c2d995"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:39.289599 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.289542 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-token" pod "b96211f7-edc8-4516-a3be-9b9f22c2d995" (UID: "b96211f7-edc8-4516-a3be-9b9f22c2d995"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.291500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.291475 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-kube-api-access-r9hjq" pod "b96211f7-edc8-4516-a3be-9b9f22c2d995" (UID: "b96211f7-edc8-4516-a3be-9b9f22c2d995"). InnerVolumeSpecName "kube-api-access-r9hjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.326782 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.326502 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:58:39.329564 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.329542 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:58:39.334975 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.334952 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:58:39.340092 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.340069 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:58:39.373141 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.373104 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.373141 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.373135 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.373141 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.373144 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.373141 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.373152 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96211f7-edc8-4516-a3be-9b9f22c2d995-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.373410 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.373161 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9hjq\" (UniqueName: \"kubernetes.io/projected/b96211f7-edc8-4516-a3be-9b9f22c2d995-kube-api-access-r9hjq\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.384475 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.384432 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:39.391475 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.391436 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:39.474725 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.474688 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-kube-api-access-qbhch\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-kube-api-access-qbhch\") pod \"bed6fc72-d294-487e-a06b-38b7ec777cff\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " Oct 30 05:58:39.474874 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.474759 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-podinfo\") pod \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " Oct 30 05:58:39.474874 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.474848 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-token\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-token\") pod \"bed6fc72-d294-487e-a06b-38b7ec777cff\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " Oct 30 05:58:39.474969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.474901 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-token\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-token\") pod \"9d986add-57ae-496e-a4d7-fdb2824f07a8\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " Oct 30 05:58:39.474969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.474936 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-kube-api-access-fqddh\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-kube-api-access-fqddh\") pod \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " Oct 30 05:58:39.475040 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475007 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d986add-57ae-496e-a4d7-fdb2824f07a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d986add-57ae-496e-a4d7-fdb2824f07a8-istiod-ca-cert\") pod \"9d986add-57ae-496e-a4d7-fdb2824f07a8\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " Oct 30 05:58:39.475071 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475055 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-data\") pod \"9d986add-57ae-496e-a4d7-fdb2824f07a8\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " Oct 30 05:58:39.475115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475097 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-certs\") pod \"bed6fc72-d294-487e-a06b-38b7ec777cff\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " Oct 30 05:58:39.475184 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475141 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-podinfo\") pod \"bed6fc72-d294-487e-a06b-38b7ec777cff\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " Oct 30 05:58:39.475250 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475218 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-data\") pod \"bed6fc72-d294-487e-a06b-38b7ec777cff\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " Oct 30 05:58:39.475300 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475249 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-certs\") pod \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " Oct 30 05:58:39.475300 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475286 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istiod-ca-cert\") pod \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " Oct 30 05:58:39.475397 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475319 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-envoy\") pod \"bed6fc72-d294-487e-a06b-38b7ec777cff\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " Oct 30 05:58:39.475397 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475357 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-envoy\") pod \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " Oct 30 05:58:39.475515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475396 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-podinfo\") pod \"9d986add-57ae-496e-a4d7-fdb2824f07a8\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " Oct 30 05:58:39.475735 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475672 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed6fc72-d294-487e-a06b-38b7ec777cff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6fc72-d294-487e-a06b-38b7ec777cff-istiod-ca-cert\") pod \"bed6fc72-d294-487e-a06b-38b7ec777cff\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " Oct 30 05:58:39.475735 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475728 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-token\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-token\") pod \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " Oct 30 05:58:39.475978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475762 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istiod-ca-cert\") pod \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " Oct 30 05:58:39.475978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475827 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-kube-api-access-ngdqg\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-kube-api-access-ngdqg\") pod \"9d986add-57ae-496e-a4d7-fdb2824f07a8\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " Oct 30 05:58:39.475978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475869 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-kube-api-access-bfkwh\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-kube-api-access-bfkwh\") pod \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " Oct 30 05:58:39.475978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475932 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-envoy\") pod \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " Oct 30 05:58:39.476203 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475982 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-socket\") pod \"bed6fc72-d294-487e-a06b-38b7ec777cff\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " Oct 30 05:58:39.477020 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.476991 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-credential-socket\") pod \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " Oct 30 05:58:39.477131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.477040 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-socket\") pod \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " Oct 30 05:58:39.477131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.477100 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-socket\") pod \"9d986add-57ae-496e-a4d7-fdb2824f07a8\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " Oct 30 05:58:39.479695 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475669 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-certs" pod "bed6fc72-d294-487e-a06b-38b7ec777cff" (UID: "bed6fc72-d294-487e-a06b-38b7ec777cff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.479814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475688 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-certs" pod "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" (UID: "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.479814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.475832 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d986add-57ae-496e-a4d7-fdb2824f07a8-istiod-ca-cert" pod "9d986add-57ae-496e-a4d7-fdb2824f07a8" (UID: "9d986add-57ae-496e-a4d7-fdb2824f07a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:39.479814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.476494 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istiod-ca-cert" pod "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" (UID: "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:39.479814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.476560 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-socket" pod "bed6fc72-d294-487e-a06b-38b7ec777cff" (UID: "bed6fc72-d294-487e-a06b-38b7ec777cff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.479814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.479383 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-data" pod "9d986add-57ae-496e-a4d7-fdb2824f07a8" (UID: "9d986add-57ae-496e-a4d7-fdb2824f07a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.479814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.479424 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed6fc72-d294-487e-a06b-38b7ec777cff-istiod-ca-cert" pod "bed6fc72-d294-487e-a06b-38b7ec777cff" (UID: "bed6fc72-d294-487e-a06b-38b7ec777cff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:39.479814 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.479720 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-credential-socket\") pod \"9d986add-57ae-496e-a4d7-fdb2824f07a8\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " Oct 30 05:58:39.480168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.479899 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-credential-socket" pod "9d986add-57ae-496e-a4d7-fdb2824f07a8" (UID: "9d986add-57ae-496e-a4d7-fdb2824f07a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.480168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.479904 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-data" pod "bed6fc72-d294-487e-a06b-38b7ec777cff" (UID: "bed6fc72-d294-487e-a06b-38b7ec777cff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.480168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.479940 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-certs\") pod \"9d986add-57ae-496e-a4d7-fdb2824f07a8\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " Oct 30 05:58:39.480168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480011 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-token\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-token\") pod \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " Oct 30 05:58:39.480168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480052 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-socket\") pod \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " Oct 30 05:58:39.480168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480083 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-data\") pod \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " Oct 30 05:58:39.480168 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480113 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-credential-socket\") pod \"bed6fc72-d294-487e-a06b-38b7ec777cff\" (UID: \"bed6fc72-d294-487e-a06b-38b7ec777cff\") " Oct 30 05:58:39.480570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480184 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-certs\") pod \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\" (UID: \"f6b46fb9-e9b9-4cc8-925f-430c5fa73259\") " Oct 30 05:58:39.480570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480215 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-envoy\") pod \"9d986add-57ae-496e-a4d7-fdb2824f07a8\" (UID: \"9d986add-57ae-496e-a4d7-fdb2824f07a8\") " Oct 30 05:58:39.480570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480253 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-data\") pod \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " Oct 30 05:58:39.480570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480291 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-credential-socket\") pod \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " Oct 30 05:58:39.480570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480298 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istiod-ca-cert" pod "f6b46fb9-e9b9-4cc8-925f-430c5fa73259" (UID: "f6b46fb9-e9b9-4cc8-925f-430c5fa73259"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:39.480570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480335 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-podinfo\") pod \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\" (UID: \"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7\") " Oct 30 05:58:39.480570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480344 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-credential-socket" pod "f6b46fb9-e9b9-4cc8-925f-430c5fa73259" (UID: "f6b46fb9-e9b9-4cc8-925f-430c5fa73259"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.480997 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.480884 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-credential-socket" pod "bed6fc72-d294-487e-a06b-38b7ec777cff" (UID: "bed6fc72-d294-487e-a06b-38b7ec777cff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.481495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481106 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-data" pod "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" (UID: "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.481495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481215 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-socket" pod "9d986add-57ae-496e-a4d7-fdb2824f07a8" (UID: "9d986add-57ae-496e-a4d7-fdb2824f07a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.481495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481300 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-certs" pod "9d986add-57ae-496e-a4d7-fdb2824f07a8" (UID: "9d986add-57ae-496e-a4d7-fdb2824f07a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.481728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481517 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-socket" pod "f6b46fb9-e9b9-4cc8-925f-430c5fa73259" (UID: "f6b46fb9-e9b9-4cc8-925f-430c5fa73259"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.481728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481695 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-data" pod "f6b46fb9-e9b9-4cc8-925f-430c5fa73259" (UID: "f6b46fb9-e9b9-4cc8-925f-430c5fa73259"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.481728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481705 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481728 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481724 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481738 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481750 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6fc72-d294-487e-a06b-38b7ec777cff-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481755 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-credential-socket" pod "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" (UID: "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481761 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481801 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481816 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481831 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481846 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481861 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481876 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481888 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481902 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481916 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d986add-57ae-496e-a4d7-fdb2824f07a8-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481929 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.481957 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.481943 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.482618 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.482156 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-kube-api-access-ngdqg" pod "9d986add-57ae-496e-a4d7-fdb2824f07a8" (UID: "9d986add-57ae-496e-a4d7-fdb2824f07a8"). InnerVolumeSpecName "kube-api-access-ngdqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.482618 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.482307 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-envoy" pod "bed6fc72-d294-487e-a06b-38b7ec777cff" (UID: "bed6fc72-d294-487e-a06b-38b7ec777cff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.482799 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.482769 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-certs" pod "f6b46fb9-e9b9-4cc8-925f-430c5fa73259" (UID: "f6b46fb9-e9b9-4cc8-925f-430c5fa73259"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.482917 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.482894 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-socket" pod "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" (UID: "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.483940 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.483839 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-kube-api-access-bfkwh" pod "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" (UID: "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7"). InnerVolumeSpecName "kube-api-access-bfkwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.484898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.484589 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-podinfo" pod "bed6fc72-d294-487e-a06b-38b7ec777cff" (UID: "bed6fc72-d294-487e-a06b-38b7ec777cff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:39.484898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.484748 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-podinfo" pod "f6b46fb9-e9b9-4cc8-925f-430c5fa73259" (UID: "f6b46fb9-e9b9-4cc8-925f-430c5fa73259"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:39.484898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.484822 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-token" pod "bed6fc72-d294-487e-a06b-38b7ec777cff" (UID: "bed6fc72-d294-487e-a06b-38b7ec777cff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.484898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.484847 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-kube-api-access-qbhch" pod "bed6fc72-d294-487e-a06b-38b7ec777cff" (UID: "bed6fc72-d294-487e-a06b-38b7ec777cff"). InnerVolumeSpecName "kube-api-access-qbhch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.484898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.484873 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-kube-api-access-fqddh" pod "f6b46fb9-e9b9-4cc8-925f-430c5fa73259" (UID: "f6b46fb9-e9b9-4cc8-925f-430c5fa73259"). InnerVolumeSpecName "kube-api-access-fqddh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.485371 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.485352 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-podinfo" pod "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" (UID: "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:39.485707 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.485684 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-envoy" pod "9d986add-57ae-496e-a4d7-fdb2824f07a8" (UID: "9d986add-57ae-496e-a4d7-fdb2824f07a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.486255 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.486233 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-envoy" pod "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" (UID: "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.486361 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.486338 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-token" pod "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" (UID: "1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.486852 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.486823 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-token" pod "f6b46fb9-e9b9-4cc8-925f-430c5fa73259" (UID: "f6b46fb9-e9b9-4cc8-925f-430c5fa73259"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.486939 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.486879 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-token" pod "9d986add-57ae-496e-a4d7-fdb2824f07a8" (UID: "9d986add-57ae-496e-a4d7-fdb2824f07a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:39.487007 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.486984 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-podinfo" pod "9d986add-57ae-496e-a4d7-fdb2824f07a8" (UID: "9d986add-57ae-496e-a4d7-fdb2824f07a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:39.487607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.487577 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-envoy" pod "f6b46fb9-e9b9-4cc8-925f-430c5fa73259" (UID: "f6b46fb9-e9b9-4cc8-925f-430c5fa73259"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:39.552341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.552251 5072 generic.go:356] "Generic (PLEG): container finished" podID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerID="ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388" exitCode=0 Oct 30 05:58:39.552341 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.552314 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" Oct 30 05:58:39.552582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.552369 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" event={"ID":"bed6fc72-d294-487e-a06b-38b7ec777cff","Type":"ContainerDied","Data":"ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388"} Oct 30 05:58:39.552582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.552424 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9" event={"ID":"bed6fc72-d294-487e-a06b-38b7ec777cff","Type":"ContainerDied","Data":"ef2d8d7050f51bee6a15b63ded990263582055ef6f0c5e0a00351d05e0ca1d4d"} Oct 30 05:58:39.552582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.552472 5072 scope.go:122] "RemoveContainer" containerID="ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388" Oct 30 05:58:39.555708 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.555677 5072 generic.go:356] "Generic (PLEG): container finished" podID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerID="28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d" exitCode=0 Oct 30 05:58:39.555868 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.555742 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" event={"ID":"9d986add-57ae-496e-a4d7-fdb2824f07a8","Type":"ContainerDied","Data":"28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d"} Oct 30 05:58:39.555868 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.555823 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" event={"ID":"9d986add-57ae-496e-a4d7-fdb2824f07a8","Type":"ContainerDied","Data":"ba2ae62a4a8cf1524b38abf8242cf12d3d9996afc0d451637be2a24fd7b066e9"} Oct 30 05:58:39.556044 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.555911 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5" Oct 30 05:58:39.568577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.565908 5072 generic.go:356] "Generic (PLEG): container finished" podID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerID="f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4" exitCode=0 Oct 30 05:58:39.568577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.566083 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" event={"ID":"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7","Type":"ContainerDied","Data":"f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4"} Oct 30 05:58:39.568577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.566131 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" event={"ID":"1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7","Type":"ContainerDied","Data":"c04f3cb34d556ecc1ec1e4d3a6ab09f7dcc678a1f6ddacbc29cf5853c8cc97b1"} Oct 30 05:58:39.568577 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.566210 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d" Oct 30 05:58:39.572989 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.572961 5072 generic.go:356] "Generic (PLEG): container finished" podID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerID="ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c" exitCode=0 Oct 30 05:58:39.573106 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.573054 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" event={"ID":"b96211f7-edc8-4516-a3be-9b9f22c2d995","Type":"ContainerDied","Data":"ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c"} Oct 30 05:58:39.573106 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.573094 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" event={"ID":"b96211f7-edc8-4516-a3be-9b9f22c2d995","Type":"ContainerDied","Data":"9fda0fda37bf13bdadf3a05e8edbd051f3ea6bbe57af1f13cd07c6310e927e5f"} Oct 30 05:58:39.573220 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.573172 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl" Oct 30 05:58:39.575052 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.575010 5072 scope.go:122] "RemoveContainer" containerID="ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388" Oct 30 05:58:39.579679 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:39.579340 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388\": container with ID starting with ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388 not found: ID does not exist" containerID="ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388" Oct 30 05:58:39.579855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.579830 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388"} err="failed to get container status \"ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388\": rpc error: code = NotFound desc = could not find container \"ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388\": container with ID starting with ec6b54b133cfa8c31ed85f2fd2fb34b45fb073845c6a5b4efc7e0c51d0a11388 not found: ID does not exist" Oct 30 05:58:39.579969 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.579958 5072 scope.go:122] "RemoveContainer" containerID="28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d" Oct 30 05:58:39.580697 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.580676 5072 generic.go:356] "Generic (PLEG): container finished" podID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerID="249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b" exitCode=0 Oct 30 05:58:39.580882 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.580735 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" Oct 30 05:58:39.581017 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.580752 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" event={"ID":"f6b46fb9-e9b9-4cc8-925f-430c5fa73259","Type":"ContainerDied","Data":"249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b"} Oct 30 05:58:39.581085 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.581056 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8" event={"ID":"f6b46fb9-e9b9-4cc8-925f-430c5fa73259","Type":"ContainerDied","Data":"e482d2f566ecc94ab03092a05d4245dda84683fc08063cefa20553153954c639"} Oct 30 05:58:39.583348 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.583325 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:39.585548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585098 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585122 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585141 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585288 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585548 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585527 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585552 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngdqg\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-kube-api-access-ngdqg\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585568 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkwh\" (UniqueName: \"kubernetes.io/projected/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-kube-api-access-bfkwh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585586 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585600 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585614 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585628 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585642 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585655 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585669 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585682 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585697 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbhch\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-kube-api-access-qbhch\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585710 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585724 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed6fc72-d294-487e-a06b-38b7ec777cff-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585740 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d986add-57ae-496e-a4d7-fdb2824f07a8-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.585752 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqddh\" (UniqueName: \"kubernetes.io/projected/f6b46fb9-e9b9-4cc8-925f-430c5fa73259-kube-api-access-fqddh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.588696 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.588674 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 05:58:39.593231 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.593214 5072 scope.go:122] "RemoveContainer" containerID="28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d" Oct 30 05:58:39.593700 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:39.593680 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d\": container with ID starting with 28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d not found: ID does not exist" containerID="28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d" Oct 30 05:58:39.593793 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.593709 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d"} err="failed to get container status \"28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d\": rpc error: code = NotFound desc = could not find container \"28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d\": container with ID starting with 28c881a695d58267586ff75cdb43b3697711af30c26a755921a19814112c174d not found: ID does not exist" Oct 30 05:58:39.593793 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.593733 5072 scope.go:122] "RemoveContainer" containerID="f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4" Oct 30 05:58:39.605224 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.605207 5072 scope.go:122] "RemoveContainer" containerID="f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4" Oct 30 05:58:39.605549 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:39.605527 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4\": container with ID starting with f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4 not found: ID does not exist" containerID="f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4" Oct 30 05:58:39.605629 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.605560 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4"} err="failed to get container status \"f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4\": rpc error: code = NotFound desc = could not find container \"f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4\": container with ID starting with f96a03699faa4aa8ce6ffb3a93f91cfacd1727ba1c4f8c9c9f89c26e3a2e38f4 not found: ID does not exist" Oct 30 05:58:39.605629 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.605580 5072 scope.go:122] "RemoveContainer" containerID="ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c" Oct 30 05:58:39.615886 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.615868 5072 scope.go:122] "RemoveContainer" containerID="ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c" Oct 30 05:58:39.616237 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:39.616211 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c\": container with ID starting with ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c not found: ID does not exist" containerID="ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c" Oct 30 05:58:39.616328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.616249 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c"} err="failed to get container status \"ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c\": rpc error: code = NotFound desc = could not find container \"ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c\": container with ID starting with ccc236c17c4f0059134fd8edcd8000d8ced64a4a7661828fc671212cbc3e1e3c not found: ID does not exist" Oct 30 05:58:39.616328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.616272 5072 scope.go:122] "RemoveContainer" containerID="249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b" Oct 30 05:58:39.626826 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.626809 5072 scope.go:122] "RemoveContainer" containerID="e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344" Oct 30 05:58:39.636075 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.636049 5072 scope.go:122] "RemoveContainer" containerID="249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b" Oct 30 05:58:39.636367 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:39.636349 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b\": container with ID starting with 249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b not found: ID does not exist" containerID="249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b" Oct 30 05:58:39.636428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.636376 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b"} err="failed to get container status \"249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b\": rpc error: code = NotFound desc = could not find container \"249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b\": container with ID starting with 249f426a0b814fab30480c12fab6d9126f7c9a0cccdfb9461bd70f4c41f7237b not found: ID does not exist" Oct 30 05:58:39.636428 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.636393 5072 scope.go:122] "RemoveContainer" containerID="e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344" Oct 30 05:58:39.636655 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:58:39.636636 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344\": container with ID starting with e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344 not found: ID does not exist" containerID="e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344" Oct 30 05:58:39.636714 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.636666 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344"} err="failed to get container status \"e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344\": rpc error: code = NotFound desc = could not find container \"e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344\": container with ID starting with e2291f75421ea26d0eedac56723d0c0b46872cd386e8d32fc0eae04c4c634344 not found: ID does not exist" Oct 30 05:58:39.734879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.734850 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8"] Oct 30 05:58:39.744041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.744013 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-pd9b8"] Oct 30 05:58:39.801896 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.801862 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9"] Oct 30 05:58:39.824917 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.824850 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-jjtw9"] Oct 30 05:58:39.868908 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.868857 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl"] Oct 30 05:58:39.874954 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.874926 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-nm6fl"] Oct 30 05:58:39.911613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.911577 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d"] Oct 30 05:58:39.915937 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.915906 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-8rd6d"] Oct 30 05:58:39.996608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:39.996567 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5"] Oct 30 05:58:40.004996 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.004964 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e94bd8-openshift-default-69fbb658bf-h6gb5"] Oct 30 05:58:40.520242 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.520211 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" path="/var/lib/kubelet/pods/1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7/volumes" Oct 30 05:58:40.520850 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.520828 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" path="/var/lib/kubelet/pods/9d986add-57ae-496e-a4d7-fdb2824f07a8/volumes" Oct 30 05:58:40.521485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.521448 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" path="/var/lib/kubelet/pods/b96211f7-edc8-4516-a3be-9b9f22c2d995/volumes" Oct 30 05:58:40.522316 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.522293 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" path="/var/lib/kubelet/pods/bed6fc72-d294-487e-a06b-38b7ec777cff/volumes" Oct 30 05:58:40.522753 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.522737 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" path="/var/lib/kubelet/pods/f6b46fb9-e9b9-4cc8-925f-430c5fa73259/volumes" Oct 30 05:58:40.572581 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.572548 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb"] Oct 30 05:58:40.573686 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573661 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" Oct 30 05:58:40.573686 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573683 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573727 5072 cpu_manager.go:414] "containerMap: removing container" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573736 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573776 5072 cpu_manager.go:414] "containerMap: removing container" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573785 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573800 5072 cpu_manager.go:414] "containerMap: removing container" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573807 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd3999c-39f6-4edf-9b57-71492639891d" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573817 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573824 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573855 5072 cpu_manager.go:414] "containerMap: removing container" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573862 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573876 5072 cpu_manager.go:414] "containerMap: removing container" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" Oct 30 05:58:40.573885 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.573884 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" Oct 30 05:58:40.574434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.574046 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" Oct 30 05:58:40.574434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.574061 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96211f7-edc8-4516-a3be-9b9f22c2d995" containerName="istio-proxy" Oct 30 05:58:40.574434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.574076 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b46fb9-e9b9-4cc8-925f-430c5fa73259" containerName="istio-proxy" Oct 30 05:58:40.574434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.574085 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed6fc72-d294-487e-a06b-38b7ec777cff" containerName="istio-proxy" Oct 30 05:58:40.574434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.574094 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d986add-57ae-496e-a4d7-fdb2824f07a8" containerName="istio-proxy" Oct 30 05:58:40.574434 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.574107 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc0f02e-06bf-4a98-bb5a-79ad67bc01f7" containerName="istio-proxy" Oct 30 05:58:40.580785 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.580765 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.596122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.595717 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39147c\"/\"kube-root-ca.crt\"" Oct 30 05:58:40.601060 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.601032 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb"] Oct 30 05:58:40.604450 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.604426 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39147c\"/\"test-model-39147c-kserve-self-signed-certs\"" Oct 30 05:58:40.604645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.604489 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39147c\"/\"default-dockercfg-pv2zw\"" Oct 30 05:58:40.604645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.604505 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39147c\"/\"openshift-service-ca.crt\"" Oct 30 05:58:40.722944 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.722913 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwgrb\" (UniqueName: \"kubernetes.io/projected/5263d63d-6812-4507-aead-a17dd19cef3b-kube-api-access-qwgrb\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.723101 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.723023 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-tmp-dir\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.723150 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.723125 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5263d63d-6812-4507-aead-a17dd19cef3b-tls-certs\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.723308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.723292 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-home\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.723400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.723389 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-dshm\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.723467 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.723440 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-model-cache\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.830540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.830336 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-tmp-dir\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.830709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.830676 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5263d63d-6812-4507-aead-a17dd19cef3b-tls-certs\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.834483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.831273 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-home\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.834483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.831311 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-dshm\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.834483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.831560 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-model-cache\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.852484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.845730 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-tmp-dir\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.852484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.850106 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwgrb\" (UniqueName: \"kubernetes.io/projected/5263d63d-6812-4507-aead-a17dd19cef3b-kube-api-access-qwgrb\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.852484 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.851201 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-home\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.876483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.852933 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-model-cache\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.876483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.867899 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-dshm\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.886483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.881096 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5263d63d-6812-4507-aead-a17dd19cef3b-tls-certs\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.902812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.892207 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwgrb\" (UniqueName: \"kubernetes.io/projected/5263d63d-6812-4507-aead-a17dd19cef3b-kube-api-access-qwgrb\") pod \"test-model-39147c-kserve-7ccd6c46df-wxbxb\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:40.902812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:40.901838 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:41.377500 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:41.377470 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb"] Oct 30 05:58:41.378651 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:58:41.378624 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5263d63d_6812_4507_aead_a17dd19cef3b.slice/crio-c6aba4e61e890f5b93426fcef1ad36789b3e651d378b884b5ced016715049e59 WatchSource:0}: Error finding container c6aba4e61e890f5b93426fcef1ad36789b3e651d378b884b5ced016715049e59: Status 404 returned error can't find the container with id c6aba4e61e890f5b93426fcef1ad36789b3e651d378b884b5ced016715049e59 Oct 30 05:58:41.609673 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:41.609633 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" event={"ID":"5263d63d-6812-4507-aead-a17dd19cef3b","Type":"ContainerStarted","Data":"ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9"} Oct 30 05:58:41.609673 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:41.609688 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" event={"ID":"5263d63d-6812-4507-aead-a17dd19cef3b","Type":"ContainerStarted","Data":"c6aba4e61e890f5b93426fcef1ad36789b3e651d378b884b5ced016715049e59"} Oct 30 05:58:41.663559 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:41.663414 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podStartSLOduration=1.663396304 podStartE2EDuration="1.663396304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:41.660730433 +0000 UTC m=+1681.264419691" watchObservedRunningTime="2026-10-30 05:58:41.663396304 +0000 UTC m=+1681.267085563" Oct 30 05:58:41.902771 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:41.902732 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:44.206328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.206292 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67"] Oct 30 05:58:44.211120 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.211099 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:58:44.240735 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.240703 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0382c2-dockercfg-r4m2v\"" Oct 30 05:58:44.240933 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.240775 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0382c2\"" Oct 30 05:58:44.248105 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.248072 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67"] Oct 30 05:58:44.254422 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.254391 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp"] Oct 30 05:58:44.259507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.259483 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:44.273533 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.273501 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f849af4d-72cb-4678-a0d9-168870a8d9c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp\" (UID: \"f849af4d-72cb-4678-a0d9-168870a8d9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:44.273671 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.273553 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4th\" (UniqueName: \"kubernetes.io/projected/a13eeada-6c59-4d2f-8aa7-1935811a334b-kube-api-access-lk4th\") pod \"payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67\" (UID: \"a13eeada-6c59-4d2f-8aa7-1935811a334b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:58:44.273727 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.273713 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13eeada-6c59-4d2f-8aa7-1935811a334b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67\" (UID: \"a13eeada-6c59-4d2f-8aa7-1935811a334b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:58:44.273841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.273824 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chp4l\" (UniqueName: \"kubernetes.io/projected/f849af4d-72cb-4678-a0d9-168870a8d9c1-kube-api-access-chp4l\") pod \"payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp\" (UID: \"f849af4d-72cb-4678-a0d9-168870a8d9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:44.284990 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.284946 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp"] Oct 30 05:58:44.385481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.381565 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f849af4d-72cb-4678-a0d9-168870a8d9c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp\" (UID: \"f849af4d-72cb-4678-a0d9-168870a8d9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:44.385481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.381620 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4th\" (UniqueName: \"kubernetes.io/projected/a13eeada-6c59-4d2f-8aa7-1935811a334b-kube-api-access-lk4th\") pod \"payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67\" (UID: \"a13eeada-6c59-4d2f-8aa7-1935811a334b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:58:44.385481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.381726 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13eeada-6c59-4d2f-8aa7-1935811a334b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67\" (UID: \"a13eeada-6c59-4d2f-8aa7-1935811a334b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:58:44.385481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.381838 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chp4l\" (UniqueName: \"kubernetes.io/projected/f849af4d-72cb-4678-a0d9-168870a8d9c1-kube-api-access-chp4l\") pod \"payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp\" (UID: \"f849af4d-72cb-4678-a0d9-168870a8d9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:44.429511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.389973 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13eeada-6c59-4d2f-8aa7-1935811a334b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67\" (UID: \"a13eeada-6c59-4d2f-8aa7-1935811a334b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:58:44.429511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.409788 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f849af4d-72cb-4678-a0d9-168870a8d9c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp\" (UID: \"f849af4d-72cb-4678-a0d9-168870a8d9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:44.429511 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.413496 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4th\" (UniqueName: \"kubernetes.io/projected/a13eeada-6c59-4d2f-8aa7-1935811a334b-kube-api-access-lk4th\") pod \"payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67\" (UID: \"a13eeada-6c59-4d2f-8aa7-1935811a334b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:58:44.481523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.479471 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chp4l\" (UniqueName: \"kubernetes.io/projected/f849af4d-72cb-4678-a0d9-168870a8d9c1-kube-api-access-chp4l\") pod \"payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp\" (UID: \"f849af4d-72cb-4678-a0d9-168870a8d9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:44.555523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.551748 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:58:44.623064 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.590134 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:44.650553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.645889 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw"] Oct 30 05:58:44.654574 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.654514 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c"] Oct 30 05:58:44.901526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.901449 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5"] Oct 30 05:58:44.908963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.908919 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:44.941925 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.941854 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5"] Oct 30 05:58:44.977103 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.977070 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:44.977433 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.977398 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:44.980659 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.980636 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:44.980858 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.980838 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:44.981026 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.981001 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:44.981212 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.981190 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:44.981335 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.981319 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:44.981529 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.981512 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpfsh\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-kube-api-access-jpfsh\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:44.981819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:44.981803 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.024945 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.024853 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-086c60-openshift-default-dockercfg-rnptw\"" Oct 30 05:58:45.083103 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083073 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083267 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083156 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083267 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083184 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083267 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083226 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083470 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083292 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083470 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083319 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083470 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083425 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083656 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083489 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpfsh\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-kube-api-access-jpfsh\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083656 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083597 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083762 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083718 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083808 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.083843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.083823 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.084069 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.084044 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.084146 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.084095 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.086177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.086150 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.087369 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.086529 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.105271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.105247 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.106174 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.106154 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpfsh\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-kube-api-access-jpfsh\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.120426 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.120405 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67"] Oct 30 05:58:45.121164 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:58:45.121141 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13eeada_6c59_4d2f_8aa7_1935811a334b.slice/crio-2d170baef5a749f9702c68dd7cb2c0ebaf866fd3f2eeb94a9f30b3379beb93ff WatchSource:0}: Error finding container 2d170baef5a749f9702c68dd7cb2c0ebaf866fd3f2eeb94a9f30b3379beb93ff: Status 404 returned error can't find the container with id 2d170baef5a749f9702c68dd7cb2c0ebaf866fd3f2eeb94a9f30b3379beb93ff Oct 30 05:58:45.222243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.222212 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp"] Oct 30 05:58:45.224743 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:58:45.224716 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf849af4d_72cb_4678_a0d9_168870a8d9c1.slice/crio-3940ac64f2d6ededf9b4fc89afb950fc0d63e596c0994caa900b22cdcffdc31e WatchSource:0}: Error finding container 3940ac64f2d6ededf9b4fc89afb950fc0d63e596c0994caa900b22cdcffdc31e: Status 404 returned error can't find the container with id 3940ac64f2d6ededf9b4fc89afb950fc0d63e596c0994caa900b22cdcffdc31e Oct 30 05:58:45.270495 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.264196 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:45.742987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.742943 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" event={"ID":"a13eeada-6c59-4d2f-8aa7-1935811a334b","Type":"ContainerStarted","Data":"2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46"} Oct 30 05:58:45.743139 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.743046 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" event={"ID":"a13eeada-6c59-4d2f-8aa7-1935811a334b","Type":"ContainerStarted","Data":"2d170baef5a749f9702c68dd7cb2c0ebaf866fd3f2eeb94a9f30b3379beb93ff"} Oct 30 05:58:45.743220 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.743195 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:58:45.745155 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.745124 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" event={"ID":"f849af4d-72cb-4678-a0d9-168870a8d9c1","Type":"ContainerStarted","Data":"cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed"} Oct 30 05:58:45.745525 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.745499 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:45.745619 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.745541 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" event={"ID":"f849af4d-72cb-4678-a0d9-168870a8d9c1","Type":"ContainerStarted","Data":"3940ac64f2d6ededf9b4fc89afb950fc0d63e596c0994caa900b22cdcffdc31e"} Oct 30 05:58:45.782401 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.782338 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" podStartSLOduration=1.78231848 podStartE2EDuration="1.78231848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:45.78205906 +0000 UTC m=+1685.385748316" watchObservedRunningTime="2026-10-30 05:58:45.78231848 +0000 UTC m=+1685.386007738" Oct 30 05:58:45.821699 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.821649 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" podStartSLOduration=1.821633971 podStartE2EDuration="1.821633971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:45.818210818 +0000 UTC m=+1685.421900077" watchObservedRunningTime="2026-10-30 05:58:45.821633971 +0000 UTC m=+1685.425323227" Oct 30 05:58:45.851440 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.851411 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5"] Oct 30 05:58:45.851979 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:58:45.851952 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffbb6af_8b6b_483c_83df_fc8fe5e9fe72.slice/crio-2469853d3f35c258e019a056fef6db2e3f013769e1340342e8be8a84ca63824f WatchSource:0}: Error finding container 2469853d3f35c258e019a056fef6db2e3f013769e1340342e8be8a84ca63824f: Status 404 returned error can't find the container with id 2469853d3f35c258e019a056fef6db2e3f013769e1340342e8be8a84ca63824f Oct 30 05:58:45.855177 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.855141 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:58:45.855251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.855221 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:58:45.855302 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:45.855267 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:58:46.755439 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:46.755400 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" event={"ID":"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72","Type":"ContainerStarted","Data":"6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48"} Oct 30 05:58:46.755898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:46.755500 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" event={"ID":"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72","Type":"ContainerStarted","Data":"2469853d3f35c258e019a056fef6db2e3f013769e1340342e8be8a84ca63824f"} Oct 30 05:58:46.810397 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:46.810333 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podStartSLOduration=2.810315824 podStartE2EDuration="2.810315824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:46.808677783 +0000 UTC m=+1686.412367041" watchObservedRunningTime="2026-10-30 05:58:46.810315824 +0000 UTC m=+1686.414005084" Oct 30 05:58:47.284710 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:47.284676 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:47.291076 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:47.291048 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:47.764269 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:47.764237 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:47.765198 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:47.765177 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 05:58:49.646424 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:49.646386 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" podUID="0e382939-b971-4260-8c87-fcdad8ec96c2" containerName="payload-processing" containerID="cri-o://2389843bd1696cc3e3440211361265aee3e0b6729cbf969d306f7fcd6a2736d1" gracePeriod=25 Oct 30 05:58:49.659330 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:49.655616 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" podUID="2f575289-1683-406b-a2c8-991d0ef9ccc4" containerName="payload-pre-processing" containerID="cri-o://41a1ad6ff1a83beff51587cbc3ba8d2538cd28430657cd2cf538a237f9c0d6ba" gracePeriod=25 Oct 30 05:58:49.782194 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:49.782162 5072 generic.go:356] "Generic (PLEG): container finished" podID="0e382939-b971-4260-8c87-fcdad8ec96c2" containerID="2389843bd1696cc3e3440211361265aee3e0b6729cbf969d306f7fcd6a2736d1" exitCode=0 Oct 30 05:58:49.782332 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:49.782231 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" event={"ID":"0e382939-b971-4260-8c87-fcdad8ec96c2","Type":"ContainerDied","Data":"2389843bd1696cc3e3440211361265aee3e0b6729cbf969d306f7fcd6a2736d1"} Oct 30 05:58:49.784034 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:49.783996 5072 generic.go:356] "Generic (PLEG): container finished" podID="2f575289-1683-406b-a2c8-991d0ef9ccc4" containerID="41a1ad6ff1a83beff51587cbc3ba8d2538cd28430657cd2cf538a237f9c0d6ba" exitCode=0 Oct 30 05:58:49.784230 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:49.784156 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" event={"ID":"2f575289-1683-406b-a2c8-991d0ef9ccc4","Type":"ContainerDied","Data":"41a1ad6ff1a83beff51587cbc3ba8d2538cd28430657cd2cf538a237f9c0d6ba"} Oct 30 05:58:49.989340 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:49.989314 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:58:50.023236 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.023197 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:58:50.133416 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.133381 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f575289-1683-406b-a2c8-991d0ef9ccc4-kube-api-access-g7v2r\" (UniqueName: \"kubernetes.io/projected/2f575289-1683-406b-a2c8-991d0ef9ccc4-kube-api-access-g7v2r\") pod \"2f575289-1683-406b-a2c8-991d0ef9ccc4\" (UID: \"2f575289-1683-406b-a2c8-991d0ef9ccc4\") " Oct 30 05:58:50.133716 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.133674 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e382939-b971-4260-8c87-fcdad8ec96c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e382939-b971-4260-8c87-fcdad8ec96c2-plugins-config-volume\") pod \"0e382939-b971-4260-8c87-fcdad8ec96c2\" (UID: \"0e382939-b971-4260-8c87-fcdad8ec96c2\") " Oct 30 05:58:50.133803 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.133779 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f575289-1683-406b-a2c8-991d0ef9ccc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f575289-1683-406b-a2c8-991d0ef9ccc4-plugins-config-volume\") pod \"2f575289-1683-406b-a2c8-991d0ef9ccc4\" (UID: \"2f575289-1683-406b-a2c8-991d0ef9ccc4\") " Oct 30 05:58:50.133897 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.133875 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e382939-b971-4260-8c87-fcdad8ec96c2-kube-api-access-twq82\" (UniqueName: \"kubernetes.io/projected/0e382939-b971-4260-8c87-fcdad8ec96c2-kube-api-access-twq82\") pod \"0e382939-b971-4260-8c87-fcdad8ec96c2\" (UID: \"0e382939-b971-4260-8c87-fcdad8ec96c2\") " Oct 30 05:58:50.134603 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.134570 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e382939-b971-4260-8c87-fcdad8ec96c2-plugins-config-volume" pod "0e382939-b971-4260-8c87-fcdad8ec96c2" (UID: "0e382939-b971-4260-8c87-fcdad8ec96c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:50.134819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.134798 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e382939-b971-4260-8c87-fcdad8ec96c2-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:50.135039 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.135013 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f575289-1683-406b-a2c8-991d0ef9ccc4-plugins-config-volume" pod "2f575289-1683-406b-a2c8-991d0ef9ccc4" (UID: "2f575289-1683-406b-a2c8-991d0ef9ccc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:50.136420 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.136399 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f575289-1683-406b-a2c8-991d0ef9ccc4-kube-api-access-g7v2r" pod "2f575289-1683-406b-a2c8-991d0ef9ccc4" (UID: "2f575289-1683-406b-a2c8-991d0ef9ccc4"). InnerVolumeSpecName "kube-api-access-g7v2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:50.137396 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.137377 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e382939-b971-4260-8c87-fcdad8ec96c2-kube-api-access-twq82" pod "0e382939-b971-4260-8c87-fcdad8ec96c2" (UID: "0e382939-b971-4260-8c87-fcdad8ec96c2"). InnerVolumeSpecName "kube-api-access-twq82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:50.236441 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.236359 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f575289-1683-406b-a2c8-991d0ef9ccc4-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:50.236441 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.236387 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twq82\" (UniqueName: \"kubernetes.io/projected/0e382939-b971-4260-8c87-fcdad8ec96c2-kube-api-access-twq82\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:50.236441 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.236398 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7v2r\" (UniqueName: \"kubernetes.io/projected/2f575289-1683-406b-a2c8-991d0ef9ccc4-kube-api-access-g7v2r\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:58:50.793895 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.793851 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" event={"ID":"0e382939-b971-4260-8c87-fcdad8ec96c2","Type":"ContainerDied","Data":"d6af83c08c2aaf7c4aa33752fefe41b4dd0c2c035853fd93935260a5f77be5f4"} Oct 30 05:58:50.794374 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.793926 5072 scope.go:122] "RemoveContainer" containerID="2389843bd1696cc3e3440211361265aee3e0b6729cbf969d306f7fcd6a2736d1" Oct 30 05:58:50.794374 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.793863 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw" Oct 30 05:58:50.795707 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.795655 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" Oct 30 05:58:50.795921 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.795702 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c" event={"ID":"2f575289-1683-406b-a2c8-991d0ef9ccc4","Type":"ContainerDied","Data":"f2bde00414e3e52a213fe98cf6451eaab72cec617ddab4bafde4785c2e21963d"} Oct 30 05:58:50.805735 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.805714 5072 scope.go:122] "RemoveContainer" containerID="41a1ad6ff1a83beff51587cbc3ba8d2538cd28430657cd2cf538a237f9c0d6ba" Oct 30 05:58:50.857362 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.857325 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw"] Oct 30 05:58:50.865262 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.865237 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76b905-59fcbc887d-8kqzw"] Oct 30 05:58:50.892065 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.892037 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c"] Oct 30 05:58:50.902926 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.898806 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76b905-77cb678995-5pq2c"] Oct 30 05:58:50.904314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.903764 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:50.925612 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:50.925584 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:51.013260 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.013221 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf"] Oct 30 05:58:51.014631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.014358 5072 cpu_manager.go:414] "containerMap: removing container" podUID="0e382939-b971-4260-8c87-fcdad8ec96c2" containerName="payload-processing" Oct 30 05:58:51.014631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.014384 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e382939-b971-4260-8c87-fcdad8ec96c2" containerName="payload-processing" Oct 30 05:58:51.014631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.014471 5072 cpu_manager.go:414] "containerMap: removing container" podUID="2f575289-1683-406b-a2c8-991d0ef9ccc4" containerName="payload-pre-processing" Oct 30 05:58:51.014631 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.014483 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f575289-1683-406b-a2c8-991d0ef9ccc4" containerName="payload-pre-processing" Oct 30 05:58:51.014930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.014654 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e382939-b971-4260-8c87-fcdad8ec96c2" containerName="payload-processing" Oct 30 05:58:51.014930 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.014683 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f575289-1683-406b-a2c8-991d0ef9ccc4" containerName="payload-pre-processing" Oct 30 05:58:51.020607 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.020584 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:58:51.045312 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.045289 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-086c60-dockercfg-zmlhj\"" Oct 30 05:58:51.045564 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.045546 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-086c60\"" Oct 30 05:58:51.046106 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.046079 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf"] Oct 30 05:58:51.063570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.063511 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7"] Oct 30 05:58:51.069353 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.069333 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:58:51.083967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.083938 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7"] Oct 30 05:58:51.246485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.236789 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blqw\" (UniqueName: \"kubernetes.io/projected/492274d9-c7ca-4530-80bf-34037bb30b9c-kube-api-access-2blqw\") pod \"payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7\" (UID: \"492274d9-c7ca-4530-80bf-34037bb30b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:58:51.246485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.237013 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08586f8a-ffbb-4bce-aecd-9a878faefbd6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf\" (UID: \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:58:51.246485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.237055 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492274d9-c7ca-4530-80bf-34037bb30b9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7\" (UID: \"492274d9-c7ca-4530-80bf-34037bb30b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:58:51.246485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.237135 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xph7b\" (UniqueName: \"kubernetes.io/projected/08586f8a-ffbb-4bce-aecd-9a878faefbd6-kube-api-access-xph7b\") pod \"payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf\" (UID: \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:58:51.345323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.344734 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08586f8a-ffbb-4bce-aecd-9a878faefbd6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf\" (UID: \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:58:51.345323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.344799 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492274d9-c7ca-4530-80bf-34037bb30b9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7\" (UID: \"492274d9-c7ca-4530-80bf-34037bb30b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:58:51.345323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.344872 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xph7b\" (UniqueName: \"kubernetes.io/projected/08586f8a-ffbb-4bce-aecd-9a878faefbd6-kube-api-access-xph7b\") pod \"payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf\" (UID: \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:58:51.345323 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.345112 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blqw\" (UniqueName: \"kubernetes.io/projected/492274d9-c7ca-4530-80bf-34037bb30b9c-kube-api-access-2blqw\") pod \"payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7\" (UID: \"492274d9-c7ca-4530-80bf-34037bb30b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:58:51.366347 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.346449 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08586f8a-ffbb-4bce-aecd-9a878faefbd6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf\" (UID: \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:58:51.366347 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.347180 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492274d9-c7ca-4530-80bf-34037bb30b9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7\" (UID: \"492274d9-c7ca-4530-80bf-34037bb30b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:58:51.442978 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.442945 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xph7b\" (UniqueName: \"kubernetes.io/projected/08586f8a-ffbb-4bce-aecd-9a878faefbd6-kube-api-access-xph7b\") pod \"payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf\" (UID: \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:58:51.461928 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.461898 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blqw\" (UniqueName: \"kubernetes.io/projected/492274d9-c7ca-4530-80bf-34037bb30b9c-kube-api-access-2blqw\") pod \"payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7\" (UID: \"492274d9-c7ca-4530-80bf-34037bb30b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:58:51.637237 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.637191 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:58:51.687209 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.687178 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:58:51.828682 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.828647 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 05:58:51.955709 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:51.955094 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf"] Oct 30 05:58:52.022394 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.022367 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7"] Oct 30 05:58:52.023914 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:58:52.023885 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492274d9_c7ca_4530_80bf_34037bb30b9c.slice/crio-707bf26ea2984dc96e283c64703000770bc2eb3b7c590b9cc5f46ddb9b28a10c WatchSource:0}: Error finding container 707bf26ea2984dc96e283c64703000770bc2eb3b7c590b9cc5f46ddb9b28a10c: Status 404 returned error can't find the container with id 707bf26ea2984dc96e283c64703000770bc2eb3b7c590b9cc5f46ddb9b28a10c Oct 30 05:58:52.563199 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.563158 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e382939-b971-4260-8c87-fcdad8ec96c2" path="/var/lib/kubelet/pods/0e382939-b971-4260-8c87-fcdad8ec96c2/volumes" Oct 30 05:58:52.564477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.564437 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f575289-1683-406b-a2c8-991d0ef9ccc4" path="/var/lib/kubelet/pods/2f575289-1683-406b-a2c8-991d0ef9ccc4/volumes" Oct 30 05:58:52.824979 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.824873 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" event={"ID":"492274d9-c7ca-4530-80bf-34037bb30b9c","Type":"ContainerStarted","Data":"b210dd9a30afe20d476fc10711cb79050ab9345f2430faffe6aaf7fa688def8d"} Oct 30 05:58:52.824979 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.824942 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" event={"ID":"492274d9-c7ca-4530-80bf-34037bb30b9c","Type":"ContainerStarted","Data":"707bf26ea2984dc96e283c64703000770bc2eb3b7c590b9cc5f46ddb9b28a10c"} Oct 30 05:58:52.824979 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.824970 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:58:52.826841 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.826811 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" event={"ID":"08586f8a-ffbb-4bce-aecd-9a878faefbd6","Type":"ContainerStarted","Data":"51bd16a41613a5910c655d916d8cad83fb17145084045be9fdbec99757b786fb"} Oct 30 05:58:52.826987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.826860 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" event={"ID":"08586f8a-ffbb-4bce-aecd-9a878faefbd6","Type":"ContainerStarted","Data":"70953831ba0666034fa4095ba70321ba7a594e26d84ce5ef502c0d15695a4dca"} Oct 30 05:58:52.826987 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.826885 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:58:52.864557 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.864505 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" podStartSLOduration=1.864491192 podStartE2EDuration="1.864491192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:52.862340576 +0000 UTC m=+1692.466029833" watchObservedRunningTime="2026-10-30 05:58:52.864491192 +0000 UTC m=+1692.468180453" Oct 30 05:58:52.901867 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:52.901813 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" podStartSLOduration=2.90179876 podStartE2EDuration="2.90179876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:52.899418502 +0000 UTC m=+1692.503107758" watchObservedRunningTime="2026-10-30 05:58:52.90179876 +0000 UTC m=+1692.505488016" Oct 30 05:58:56.759255 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:56.759225 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 05:58:56.759810 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:58:56.759306 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 05:59:00.719503 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:59:00.718717 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:59:03.838594 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:03.838568 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 05:59:03.838985 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:03.838631 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 05:59:04.802477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:04.802432 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd"] Oct 30 05:59:04.802780 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:04.802752 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" containerID="cri-o://60fab8e1061bdc157d798e5b2832a17b701bf8c917e7da167cefc9792a3b0df6" gracePeriod=30 Oct 30 05:59:09.612593 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.612553 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x"] Oct 30 05:59:09.626943 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.626894 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.644408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.644372 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.645160 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.645135 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-metrics-certs\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.645366 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.645344 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-maas-api-tls\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.645535 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.645516 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8glp\" (UniqueName: \"kubernetes.io/projected/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-kube-api-access-g8glp\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.656564 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.655853 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x"] Oct 30 05:59:09.657625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.657599 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0382c2-serving-cert\"" Oct 30 05:59:09.747530 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.747489 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-metrics-certs\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.747721 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.747570 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-maas-api-tls\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.747721 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.747612 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8glp\" (UniqueName: \"kubernetes.io/projected/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-kube-api-access-g8glp\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.747892 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.747872 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.748662 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.748634 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.750344 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.750325 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-maas-api-tls\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.750422 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.750340 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-metrics-certs\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.770051 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.770022 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8glp\" (UniqueName: \"kubernetes.io/projected/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-kube-api-access-g8glp\") pod \"maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:09.989081 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:09.988992 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:10.099082 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.099043 5072 generic.go:356] "Generic (PLEG): container finished" podID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerID="60fab8e1061bdc157d798e5b2832a17b701bf8c917e7da167cefc9792a3b0df6" exitCode=0 Oct 30 05:59:10.099258 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.099235 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" event={"ID":"98048f3c-0f13-4c88-b03a-49a0d055e156","Type":"ContainerDied","Data":"60fab8e1061bdc157d798e5b2832a17b701bf8c917e7da167cefc9792a3b0df6"} Oct 30 05:59:10.270245 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.270218 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:59:10.336411 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.336371 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x"] Oct 30 05:59:10.347106 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:59:10.347075 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711d1d7d_61f0_4ad6_bfd7_64bdf2ad593a.slice/crio-c3d9c45fd6fe16bab095d0374e5cbb2f6de9cda864b8de9f73a6cdaea07c045d WatchSource:0}: Error finding container c3d9c45fd6fe16bab095d0374e5cbb2f6de9cda864b8de9f73a6cdaea07c045d: Status 404 returned error can't find the container with id c3d9c45fd6fe16bab095d0374e5cbb2f6de9cda864b8de9f73a6cdaea07c045d Oct 30 05:59:10.375694 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.375667 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-credential-socket\") pod \"98048f3c-0f13-4c88-b03a-49a0d055e156\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " Oct 30 05:59:10.375856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.375752 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-socket\") pod \"98048f3c-0f13-4c88-b03a-49a0d055e156\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " Oct 30 05:59:10.375856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.375805 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98048f3c-0f13-4c88-b03a-49a0d055e156-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98048f3c-0f13-4c88-b03a-49a0d055e156-istiod-ca-cert\") pod \"98048f3c-0f13-4c88-b03a-49a0d055e156\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " Oct 30 05:59:10.375856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.375832 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-kube-api-access-256lg\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-kube-api-access-256lg\") pod \"98048f3c-0f13-4c88-b03a-49a0d055e156\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " Oct 30 05:59:10.376038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.375900 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-podinfo\") pod \"98048f3c-0f13-4c88-b03a-49a0d055e156\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " Oct 30 05:59:10.376038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.375983 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-data\") pod \"98048f3c-0f13-4c88-b03a-49a0d055e156\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " Oct 30 05:59:10.376038 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.376018 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-credential-socket" pod "98048f3c-0f13-4c88-b03a-49a0d055e156" (UID: "98048f3c-0f13-4c88-b03a-49a0d055e156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.376209 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.376059 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-envoy\") pod \"98048f3c-0f13-4c88-b03a-49a0d055e156\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " Oct 30 05:59:10.376209 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.376103 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-socket" pod "98048f3c-0f13-4c88-b03a-49a0d055e156" (UID: "98048f3c-0f13-4c88-b03a-49a0d055e156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.376209 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.376111 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-token\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-token\") pod \"98048f3c-0f13-4c88-b03a-49a0d055e156\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " Oct 30 05:59:10.376355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.376315 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-certs\") pod \"98048f3c-0f13-4c88-b03a-49a0d055e156\" (UID: \"98048f3c-0f13-4c88-b03a-49a0d055e156\") " Oct 30 05:59:10.376470 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.376431 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-data" pod "98048f3c-0f13-4c88-b03a-49a0d055e156" (UID: "98048f3c-0f13-4c88-b03a-49a0d055e156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.376940 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.376853 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-certs" pod "98048f3c-0f13-4c88-b03a-49a0d055e156" (UID: "98048f3c-0f13-4c88-b03a-49a0d055e156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.377045 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.377023 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.377098 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.377043 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.377098 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.377056 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.377208 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.377115 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98048f3c-0f13-4c88-b03a-49a0d055e156-istiod-ca-cert" pod "98048f3c-0f13-4c88-b03a-49a0d055e156" (UID: "98048f3c-0f13-4c88-b03a-49a0d055e156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:10.379019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.378992 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-envoy" pod "98048f3c-0f13-4c88-b03a-49a0d055e156" (UID: "98048f3c-0f13-4c88-b03a-49a0d055e156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.379180 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.379162 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-podinfo" pod "98048f3c-0f13-4c88-b03a-49a0d055e156" (UID: "98048f3c-0f13-4c88-b03a-49a0d055e156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:10.379745 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.379721 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-kube-api-access-256lg" pod "98048f3c-0f13-4c88-b03a-49a0d055e156" (UID: "98048f3c-0f13-4c88-b03a-49a0d055e156"). InnerVolumeSpecName "kube-api-access-256lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:10.379862 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.379849 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-token" pod "98048f3c-0f13-4c88-b03a-49a0d055e156" (UID: "98048f3c-0f13-4c88-b03a-49a0d055e156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:10.478791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.478758 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98048f3c-0f13-4c88-b03a-49a0d055e156-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.478791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.478785 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-256lg\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-kube-api-access-256lg\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.478791 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.478795 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.479128 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.478805 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.479128 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.478814 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98048f3c-0f13-4c88-b03a-49a0d055e156-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.479128 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:10.478822 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98048f3c-0f13-4c88-b03a-49a0d055e156-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:11.114408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:11.114323 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" event={"ID":"98048f3c-0f13-4c88-b03a-49a0d055e156","Type":"ContainerDied","Data":"c402cd710c1f5a5990a375b10d127bcd35bb0c9b46ac2ad3f00d8c2be28736cf"} Oct 30 05:59:11.114408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:11.114397 5072 scope.go:122] "RemoveContainer" containerID="60fab8e1061bdc157d798e5b2832a17b701bf8c917e7da167cefc9792a3b0df6" Oct 30 05:59:11.114951 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:11.114572 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd" Oct 30 05:59:11.118675 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:11.118627 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" event={"ID":"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a","Type":"ContainerStarted","Data":"7d65256d7e012cef0179aa8df04c029727800d96015a5d1ba269ffc6f913d833"} Oct 30 05:59:11.118863 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:11.118688 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" event={"ID":"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a","Type":"ContainerStarted","Data":"c3d9c45fd6fe16bab095d0374e5cbb2f6de9cda864b8de9f73a6cdaea07c045d"} Oct 30 05:59:11.119536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:11.119517 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:11.178646 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:11.178592 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" podStartSLOduration=1.7646711320000001 podStartE2EDuration="2.178576625s" totalImagesPullingTime="413.905493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:11.177758117 +0000 UTC m=+1710.781447368" watchObservedRunningTime="2026-10-30 05:59:11.178576625 +0000 UTC m=+1710.782265882" Oct 30 05:59:11.216328 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:11.216292 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd"] Oct 30 05:59:11.227012 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:11.226975 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-t4cjd"] Oct 30 05:59:12.523563 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:12.523526 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" path="/var/lib/kubelet/pods/98048f3c-0f13-4c88-b03a-49a0d055e156/volumes" Oct 30 05:59:18.147837 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.147808 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 05:59:18.384866 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.384832 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2"] Oct 30 05:59:18.385608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.385593 5072 cpu_manager.go:414] "containerMap: removing container" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" Oct 30 05:59:18.385673 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.385608 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" Oct 30 05:59:18.385762 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.385753 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="98048f3c-0f13-4c88-b03a-49a0d055e156" containerName="istio-proxy" Oct 30 05:59:18.391546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.391523 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.444396 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.444365 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-086c60-serving-cert\"" Oct 30 05:59:18.452767 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.452741 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2"] Oct 30 05:59:18.561890 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.561846 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b8be3-8690-4f96-a0e1-223a035ea62e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.562062 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.561978 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-metrics-certs\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.562062 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.562016 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfznq\" (UniqueName: \"kubernetes.io/projected/d66b8be3-8690-4f96-a0e1-223a035ea62e-kube-api-access-qfznq\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.562183 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.562103 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-maas-api-tls\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.707528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.677609 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-maas-api-tls\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.707528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.678081 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b8be3-8690-4f96-a0e1-223a035ea62e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.707528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.678161 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-metrics-certs\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.707528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.678187 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfznq\" (UniqueName: \"kubernetes.io/projected/d66b8be3-8690-4f96-a0e1-223a035ea62e-kube-api-access-qfznq\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.707528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.681377 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b8be3-8690-4f96-a0e1-223a035ea62e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.707528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.694090 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-maas-api-tls\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.714048 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.709187 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-metrics-certs\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.714048 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.712881 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfznq\" (UniqueName: \"kubernetes.io/projected/d66b8be3-8690-4f96-a0e1-223a035ea62e-kube-api-access-qfznq\") pod \"maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:18.751209 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:18.751179 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:19.066304 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:19.066275 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2"] Oct 30 05:59:19.067342 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:59:19.067316 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66b8be3_8690_4f96_a0e1_223a035ea62e.slice/crio-4a2847215c829029ae7c67b51afbcc2c4f4b5c0aa605ce8622c53c6b67de1967 WatchSource:0}: Error finding container 4a2847215c829029ae7c67b51afbcc2c4f4b5c0aa605ce8622c53c6b67de1967: Status 404 returned error can't find the container with id 4a2847215c829029ae7c67b51afbcc2c4f4b5c0aa605ce8622c53c6b67de1967 Oct 30 05:59:19.194096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:19.194056 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" event={"ID":"d66b8be3-8690-4f96-a0e1-223a035ea62e","Type":"ContainerStarted","Data":"4a2847215c829029ae7c67b51afbcc2c4f4b5c0aa605ce8622c53c6b67de1967"} Oct 30 05:59:20.203798 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:20.203764 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" event={"ID":"d66b8be3-8690-4f96-a0e1-223a035ea62e","Type":"ContainerStarted","Data":"147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a"} Oct 30 05:59:20.204291 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:20.203862 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:20.245536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:20.245480 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" podStartSLOduration=1.9306168129999999 podStartE2EDuration="2.245465329s" totalImagesPullingTime="314.848516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:20.241845135 +0000 UTC m=+1719.845534392" watchObservedRunningTime="2026-10-30 05:59:20.245465329 +0000 UTC m=+1719.849154576" Oct 30 05:59:26.217556 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:26.217525 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 05:59:30.712833 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:59:30.712784 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 05:59:31.954517 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:31.954479 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh"] Oct 30 05:59:31.955058 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:31.954726 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" containerID="cri-o://693efd0944f009f952915003933de8b80ced4e44528e394b357eab150dc5f7a3" gracePeriod=30 Oct 30 05:59:31.963935 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:31.963902 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss"] Oct 30 05:59:31.964130 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:31.964110 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" containerID="cri-o://f637cd64b1caf3ffc98fe178cad824370ec08a99e59dfd43b1a8145125fc073a" gracePeriod=30 Oct 30 05:59:32.131102 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:32.131070 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft"] Oct 30 05:59:32.131358 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:32.131325 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" podUID="f2d5902b-5ff6-4217-a48e-33a0bf5056aa" containerName="maas-api" containerID="cri-o://2915c29dbf333c8929fb9c84575bd9f0bae05a202cd312200027be9cd277aa2d" gracePeriod=30 Oct 30 05:59:32.367223 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:32.367185 5072 generic.go:356] "Generic (PLEG): container finished" podID="f2d5902b-5ff6-4217-a48e-33a0bf5056aa" containerID="2915c29dbf333c8929fb9c84575bd9f0bae05a202cd312200027be9cd277aa2d" exitCode=0 Oct 30 05:59:32.367405 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:32.367256 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" event={"ID":"f2d5902b-5ff6-4217-a48e-33a0bf5056aa","Type":"ContainerDied","Data":"2915c29dbf333c8929fb9c84575bd9f0bae05a202cd312200027be9cd277aa2d"} Oct 30 05:59:32.605166 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:32.605060 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:59:32.886578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:32.868556 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-kube-api-access-b6fp7\" (UniqueName: \"kubernetes.io/projected/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-kube-api-access-b6fp7\") pod \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " Oct 30 05:59:32.886578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:32.869025 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-openshift-service-ca\") pod \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " Oct 30 05:59:32.886578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:32.869081 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-maas-api-tls\") pod \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " Oct 30 05:59:32.886578 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:32.869141 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-metrics-certs\") pod \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\" (UID: \"f2d5902b-5ff6-4217-a48e-33a0bf5056aa\") " Oct 30 05:59:33.014294 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.006619 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-openshift-service-ca" pod "f2d5902b-5ff6-4217-a48e-33a0bf5056aa" (UID: "f2d5902b-5ff6-4217-a48e-33a0bf5056aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:33.066485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.039493 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:33.070938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.070895 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-metrics-certs" pod "f2d5902b-5ff6-4217-a48e-33a0bf5056aa" (UID: "f2d5902b-5ff6-4217-a48e-33a0bf5056aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:33.075491 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.074907 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-maas-api-tls" pod "f2d5902b-5ff6-4217-a48e-33a0bf5056aa" (UID: "f2d5902b-5ff6-4217-a48e-33a0bf5056aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:33.078627 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.078600 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-kube-api-access-b6fp7" pod "f2d5902b-5ff6-4217-a48e-33a0bf5056aa" (UID: "f2d5902b-5ff6-4217-a48e-33a0bf5056aa"). InnerVolumeSpecName "kube-api-access-b6fp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:33.141377 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.141299 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:33.141377 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.141335 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:33.141377 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.141350 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6fp7\" (UniqueName: \"kubernetes.io/projected/f2d5902b-5ff6-4217-a48e-33a0bf5056aa-kube-api-access-b6fp7\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:33.404542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.402007 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" event={"ID":"f2d5902b-5ff6-4217-a48e-33a0bf5056aa","Type":"ContainerDied","Data":"cd2369fcc594b45cb01c06c363f42b676b294e07ebca07b53194a8a452947544"} Oct 30 05:59:33.404542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.402105 5072 scope.go:122] "RemoveContainer" containerID="2915c29dbf333c8929fb9c84575bd9f0bae05a202cd312200027be9cd277aa2d" Oct 30 05:59:33.404542 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.402561 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft" Oct 30 05:59:33.465376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.465340 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft"] Oct 30 05:59:33.475055 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:33.475027 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-530a5e-795dfd6786-bwvft"] Oct 30 05:59:34.527493 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:34.525374 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d5902b-5ff6-4217-a48e-33a0bf5056aa" path="/var/lib/kubelet/pods/f2d5902b-5ff6-4217-a48e-33a0bf5056aa/volumes" Oct 30 05:59:35.408964 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.408915 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l"] Oct 30 05:59:35.410140 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.410115 5072 cpu_manager.go:414] "containerMap: removing container" podUID="f2d5902b-5ff6-4217-a48e-33a0bf5056aa" containerName="maas-api" Oct 30 05:59:35.410140 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.410139 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d5902b-5ff6-4217-a48e-33a0bf5056aa" containerName="maas-api" Oct 30 05:59:35.410355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.410343 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d5902b-5ff6-4217-a48e-33a0bf5056aa" containerName="maas-api" Oct 30 05:59:35.415823 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.415803 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.440516 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.440487 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l"] Oct 30 05:59:35.444473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.444424 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b3e625-openshift-default-dockercfg-xfmdc\"" Oct 30 05:59:35.502812 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.502781 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-envoy\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.502997 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.502827 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-token\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.502997 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.502856 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-certs\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.503083 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.503055 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-data\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.503190 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.503173 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-podinfo\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.503235 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.503220 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-credential-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.503306 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.503292 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vvx\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-kube-api-access-44vvx\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.503363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.503328 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istiod-ca-cert\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.503430 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.503418 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.604843 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.604807 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-token\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.604869 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-certs\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.604994 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-data\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605095 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-podinfo\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605123 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-credential-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605233 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vvx\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-kube-api-access-44vvx\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605255 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-certs\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605285 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istiod-ca-cert\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605683 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605354 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605683 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605415 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-envoy\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605683 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605354 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-data\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605683 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605504 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-credential-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605683 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605643 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.605903 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.605886 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istiod-ca-cert\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.607856 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.607831 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-podinfo\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.608018 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.607839 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-envoy\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.627338 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.627313 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-token\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.627633 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.627610 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vvx\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-kube-api-access-44vvx\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:35.732044 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:35.731958 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:36.060445 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:36.060420 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l"] Oct 30 05:59:36.061135 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:59:36.061109 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6da94b_aa21_4e3b_9219_35fac0bc7f3d.slice/crio-0eb53270428d56a7f9786698573891642e62928b428e46ecd3d401d7da32efe1 WatchSource:0}: Error finding container 0eb53270428d56a7f9786698573891642e62928b428e46ecd3d401d7da32efe1: Status 404 returned error can't find the container with id 0eb53270428d56a7f9786698573891642e62928b428e46ecd3d401d7da32efe1 Oct 30 05:59:36.064295 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:36.064259 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:36.064377 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:36.064351 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:36.064421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:36.064380 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:36.435025 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:36.434987 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" event={"ID":"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d","Type":"ContainerStarted","Data":"6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08"} Oct 30 05:59:36.435238 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:36.435059 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" event={"ID":"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d","Type":"ContainerStarted","Data":"0eb53270428d56a7f9786698573891642e62928b428e46ecd3d401d7da32efe1"} Oct 30 05:59:36.526091 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:36.526029 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" podStartSLOduration=1.5260080999999999 podStartE2EDuration="1.5260081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:36.512317009 +0000 UTC m=+1736.116006274" watchObservedRunningTime="2026-10-30 05:59:36.5260081 +0000 UTC m=+1736.129697361" Oct 30 05:59:36.732834 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:36.732745 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:37.475836 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.475807 5072 generic.go:356] "Generic (PLEG): container finished" podID="25d474a0-f643-48e6-b1e3-f97070a90298" containerID="f637cd64b1caf3ffc98fe178cad824370ec08a99e59dfd43b1a8145125fc073a" exitCode=0 Oct 30 05:59:37.475925 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.475882 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" event={"ID":"25d474a0-f643-48e6-b1e3-f97070a90298","Type":"ContainerDied","Data":"f637cd64b1caf3ffc98fe178cad824370ec08a99e59dfd43b1a8145125fc073a"} Oct 30 05:59:37.477519 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.477499 5072 generic.go:356] "Generic (PLEG): container finished" podID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerID="693efd0944f009f952915003933de8b80ced4e44528e394b357eab150dc5f7a3" exitCode=0 Oct 30 05:59:37.477738 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.477713 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" event={"ID":"45652c65-84b3-4463-92fe-9bb9a8e230f7","Type":"ContainerDied","Data":"693efd0944f009f952915003933de8b80ced4e44528e394b357eab150dc5f7a3"} Oct 30 05:59:37.477822 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.477786 5072 scope.go:122] "RemoveContainer" containerID="fcb861e348d08f9bc45bacd3b6e51b0a9388bb2916acf9d6de2440c7e4b5b185" Oct 30 05:59:37.504901 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.504874 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:59:37.508119 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.508096 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:59:37.584096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.583842 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-socket\") pod \"45652c65-84b3-4463-92fe-9bb9a8e230f7\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " Oct 30 05:59:37.584096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.583898 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-kube-api-access-2td2w\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-kube-api-access-2td2w\") pod \"45652c65-84b3-4463-92fe-9bb9a8e230f7\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " Oct 30 05:59:37.584096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.583955 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-podinfo\") pod \"45652c65-84b3-4463-92fe-9bb9a8e230f7\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " Oct 30 05:59:37.584096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.583994 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-credential-socket\") pod \"45652c65-84b3-4463-92fe-9bb9a8e230f7\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " Oct 30 05:59:37.584096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584026 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-kube-api-access-4rjhh\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-kube-api-access-4rjhh\") pod \"25d474a0-f643-48e6-b1e3-f97070a90298\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " Oct 30 05:59:37.584096 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584075 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-certs\") pod \"45652c65-84b3-4463-92fe-9bb9a8e230f7\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " Oct 30 05:59:37.584582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584141 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-socket" pod "45652c65-84b3-4463-92fe-9bb9a8e230f7" (UID: "45652c65-84b3-4463-92fe-9bb9a8e230f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.584582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584154 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-envoy\") pod \"25d474a0-f643-48e6-b1e3-f97070a90298\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " Oct 30 05:59:37.584582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584292 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d474a0-f643-48e6-b1e3-f97070a90298-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d474a0-f643-48e6-b1e3-f97070a90298-istio-podinfo\") pod \"25d474a0-f643-48e6-b1e3-f97070a90298\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " Oct 30 05:59:37.584582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584371 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-data\") pod \"45652c65-84b3-4463-92fe-9bb9a8e230f7\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " Oct 30 05:59:37.584582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584511 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45652c65-84b3-4463-92fe-9bb9a8e230f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45652c65-84b3-4463-92fe-9bb9a8e230f7-istiod-ca-cert\") pod \"45652c65-84b3-4463-92fe-9bb9a8e230f7\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " Oct 30 05:59:37.584582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584547 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-credential-socket\") pod \"25d474a0-f643-48e6-b1e3-f97070a90298\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " Oct 30 05:59:37.584582 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584577 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-token\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-token\") pod \"45652c65-84b3-4463-92fe-9bb9a8e230f7\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " Oct 30 05:59:37.584962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584617 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d474a0-f643-48e6-b1e3-f97070a90298-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d474a0-f643-48e6-b1e3-f97070a90298-istiod-ca-cert\") pod \"25d474a0-f643-48e6-b1e3-f97070a90298\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " Oct 30 05:59:37.584962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584649 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-istio-token\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-istio-token\") pod \"25d474a0-f643-48e6-b1e3-f97070a90298\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " Oct 30 05:59:37.584962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584680 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-certs\") pod \"25d474a0-f643-48e6-b1e3-f97070a90298\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " Oct 30 05:59:37.584962 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.584796 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-data\") pod \"25d474a0-f643-48e6-b1e3-f97070a90298\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " Oct 30 05:59:37.585173 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.585014 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-socket\") pod \"25d474a0-f643-48e6-b1e3-f97070a90298\" (UID: \"25d474a0-f643-48e6-b1e3-f97070a90298\") " Oct 30 05:59:37.585173 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.585048 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-envoy\") pod \"45652c65-84b3-4463-92fe-9bb9a8e230f7\" (UID: \"45652c65-84b3-4463-92fe-9bb9a8e230f7\") " Oct 30 05:59:37.585855 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.585570 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-data" pod "45652c65-84b3-4463-92fe-9bb9a8e230f7" (UID: "45652c65-84b3-4463-92fe-9bb9a8e230f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.586015 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.585991 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.586080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.586022 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.587264 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.587223 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-kube-api-access-2td2w" pod "45652c65-84b3-4463-92fe-9bb9a8e230f7" (UID: "45652c65-84b3-4463-92fe-9bb9a8e230f7"). InnerVolumeSpecName "kube-api-access-2td2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:37.587733 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.587717 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-certs" pod "25d474a0-f643-48e6-b1e3-f97070a90298" (UID: "25d474a0-f643-48e6-b1e3-f97070a90298"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.587819 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.587736 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-socket" pod "25d474a0-f643-48e6-b1e3-f97070a90298" (UID: "25d474a0-f643-48e6-b1e3-f97070a90298"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.587907 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.587757 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45652c65-84b3-4463-92fe-9bb9a8e230f7-istiod-ca-cert" pod "45652c65-84b3-4463-92fe-9bb9a8e230f7" (UID: "45652c65-84b3-4463-92fe-9bb9a8e230f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:37.587959 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.587934 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-data" pod "25d474a0-f643-48e6-b1e3-f97070a90298" (UID: "25d474a0-f643-48e6-b1e3-f97070a90298"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.588224 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.588207 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-credential-socket" pod "45652c65-84b3-4463-92fe-9bb9a8e230f7" (UID: "45652c65-84b3-4463-92fe-9bb9a8e230f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.588350 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.588324 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d474a0-f643-48e6-b1e3-f97070a90298-istiod-ca-cert" pod "25d474a0-f643-48e6-b1e3-f97070a90298" (UID: "25d474a0-f643-48e6-b1e3-f97070a90298"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:37.588815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.588795 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-certs" pod "45652c65-84b3-4463-92fe-9bb9a8e230f7" (UID: "45652c65-84b3-4463-92fe-9bb9a8e230f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.589547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.588995 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-podinfo" pod "45652c65-84b3-4463-92fe-9bb9a8e230f7" (UID: "45652c65-84b3-4463-92fe-9bb9a8e230f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:37.589547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.589063 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-token" pod "45652c65-84b3-4463-92fe-9bb9a8e230f7" (UID: "45652c65-84b3-4463-92fe-9bb9a8e230f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:37.589547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.589356 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-envoy" pod "25d474a0-f643-48e6-b1e3-f97070a90298" (UID: "25d474a0-f643-48e6-b1e3-f97070a90298"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.589547 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.589383 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d474a0-f643-48e6-b1e3-f97070a90298-istio-podinfo" pod "25d474a0-f643-48e6-b1e3-f97070a90298" (UID: "25d474a0-f643-48e6-b1e3-f97070a90298"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:37.590065 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.590046 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-credential-socket" pod "25d474a0-f643-48e6-b1e3-f97070a90298" (UID: "25d474a0-f643-48e6-b1e3-f97070a90298"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.590363 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.590343 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-istio-token" pod "25d474a0-f643-48e6-b1e3-f97070a90298" (UID: "25d474a0-f643-48e6-b1e3-f97070a90298"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:37.590587 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.590572 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-envoy" pod "45652c65-84b3-4463-92fe-9bb9a8e230f7" (UID: "45652c65-84b3-4463-92fe-9bb9a8e230f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:37.590761 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.590741 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-kube-api-access-4rjhh" pod "25d474a0-f643-48e6-b1e3-f97070a90298" (UID: "25d474a0-f643-48e6-b1e3-f97070a90298"). InnerVolumeSpecName "kube-api-access-4rjhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688606 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d474a0-f643-48e6-b1e3-f97070a90298-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688653 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45652c65-84b3-4463-92fe-9bb9a8e230f7-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688669 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688683 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688710 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d474a0-f643-48e6-b1e3-f97070a90298-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688723 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688735 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688747 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688761 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688786 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688801 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2td2w\" (UniqueName: \"kubernetes.io/projected/45652c65-84b3-4463-92fe-9bb9a8e230f7-kube-api-access-2td2w\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688814 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45652c65-84b3-4463-92fe-9bb9a8e230f7-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688827 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688840 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rjhh\" (UniqueName: \"kubernetes.io/projected/25d474a0-f643-48e6-b1e3-f97070a90298-kube-api-access-4rjhh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688867 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45652c65-84b3-4463-92fe-9bb9a8e230f7-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.690115 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.688880 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d474a0-f643-48e6-b1e3-f97070a90298-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:37.770482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.761813 5072 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 05:59:37.770482 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:37.761895 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 05:59:38.256644 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.245017 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz"] Oct 30 05:59:38.256644 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.245411 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" podUID="2423923f-15e4-4204-86de-5a6483ee8356" containerName="maas-api" containerID="cri-o://e582570eff99ba2387b5a242acc45524dce24450dba0d1535b231d7741338046" gracePeriod=30 Oct 30 05:59:38.488605 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.488569 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" event={"ID":"25d474a0-f643-48e6-b1e3-f97070a90298","Type":"ContainerDied","Data":"d1748d05c27e37dbcf5a62b2dfde55637f7115779cdca50a3368fb07482ec95e"} Oct 30 05:59:38.488705 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.488680 5072 scope.go:122] "RemoveContainer" containerID="f637cd64b1caf3ffc98fe178cad824370ec08a99e59dfd43b1a8145125fc073a" Oct 30 05:59:38.488864 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.488850 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss" Oct 30 05:59:38.547693 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.496134 5072 generic.go:356] "Generic (PLEG): container finished" podID="2423923f-15e4-4204-86de-5a6483ee8356" containerID="e582570eff99ba2387b5a242acc45524dce24450dba0d1535b231d7741338046" exitCode=0 Oct 30 05:59:38.547693 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.496274 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" event={"ID":"2423923f-15e4-4204-86de-5a6483ee8356","Type":"ContainerDied","Data":"e582570eff99ba2387b5a242acc45524dce24450dba0d1535b231d7741338046"} Oct 30 05:59:38.547693 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.499825 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" Oct 30 05:59:38.547693 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.520867 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh" event={"ID":"45652c65-84b3-4463-92fe-9bb9a8e230f7","Type":"ContainerDied","Data":"d711cc4ca81b6a4c465f8968679aa6831670539de31f905c57853174f220abee"} Oct 30 05:59:38.697637 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.696507 5072 scope.go:122] "RemoveContainer" containerID="693efd0944f009f952915003933de8b80ced4e44528e394b357eab150dc5f7a3" Oct 30 05:59:38.803537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.803126 5072 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 05:59:38.803537 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.803182 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 05:59:38.987775 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:38.987638 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:59:39.141881 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.120033 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2423923f-15e4-4204-86de-5a6483ee8356-kube-api-access-fq2nz\" (UniqueName: \"kubernetes.io/projected/2423923f-15e4-4204-86de-5a6483ee8356-kube-api-access-fq2nz\") pod \"2423923f-15e4-4204-86de-5a6483ee8356\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " Oct 30 05:59:39.141881 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.120193 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-maas-api-tls\") pod \"2423923f-15e4-4204-86de-5a6483ee8356\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " Oct 30 05:59:39.141881 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.120226 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-metrics-certs\") pod \"2423923f-15e4-4204-86de-5a6483ee8356\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " Oct 30 05:59:39.141881 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.120287 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2423923f-15e4-4204-86de-5a6483ee8356-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423923f-15e4-4204-86de-5a6483ee8356-openshift-service-ca\") pod \"2423923f-15e4-4204-86de-5a6483ee8356\" (UID: \"2423923f-15e4-4204-86de-5a6483ee8356\") " Oct 30 05:59:39.182743 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.182693 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2423923f-15e4-4204-86de-5a6483ee8356-openshift-service-ca" pod "2423923f-15e4-4204-86de-5a6483ee8356" (UID: "2423923f-15e4-4204-86de-5a6483ee8356"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:39.192421 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.192385 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-metrics-certs" pod "2423923f-15e4-4204-86de-5a6483ee8356" (UID: "2423923f-15e4-4204-86de-5a6483ee8356"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:39.193122 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.193096 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-maas-api-tls" pod "2423923f-15e4-4204-86de-5a6483ee8356" (UID: "2423923f-15e4-4204-86de-5a6483ee8356"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:39.276620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.258251 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2423923f-15e4-4204-86de-5a6483ee8356-kube-api-access-fq2nz" pod "2423923f-15e4-4204-86de-5a6483ee8356" (UID: "2423923f-15e4-4204-86de-5a6483ee8356"). InnerVolumeSpecName "kube-api-access-fq2nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:39.276620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.274579 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.276620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.274612 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2423923f-15e4-4204-86de-5a6483ee8356-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.276620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.274627 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423923f-15e4-4204-86de-5a6483ee8356-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.276620 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.274642 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq2nz\" (UniqueName: \"kubernetes.io/projected/2423923f-15e4-4204-86de-5a6483ee8356-kube-api-access-fq2nz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.281298 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.281267 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss"] Oct 30 05:59:39.281524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.281509 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ckzss"] Oct 30 05:59:39.283045 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.283029 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh"] Oct 30 05:59:39.286647 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.286521 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-l5hdh"] Oct 30 05:59:39.604536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.602160 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" event={"ID":"2423923f-15e4-4204-86de-5a6483ee8356","Type":"ContainerDied","Data":"04703b4bfc88cd19643f44783cc6a7d2b995b6345bd4f5afb0a810e01fa7a457"} Oct 30 05:59:39.604536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.602230 5072 scope.go:122] "RemoveContainer" containerID="e582570eff99ba2387b5a242acc45524dce24450dba0d1535b231d7741338046" Oct 30 05:59:39.604536 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.602701 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz" Oct 30 05:59:39.734402 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.734365 5072 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 05:59:39.734714 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.734682 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 05:59:39.762608 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.761824 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz"] Oct 30 05:59:39.765858 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:39.765831 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4323f6-6bdbf5595c-58tpz"] Oct 30 05:59:40.555992 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.555963 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2423923f-15e4-4204-86de-5a6483ee8356" path="/var/lib/kubelet/pods/2423923f-15e4-4204-86de-5a6483ee8356/volumes" Oct 30 05:59:40.556546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.556527 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" path="/var/lib/kubelet/pods/25d474a0-f643-48e6-b1e3-f97070a90298/volumes" Oct 30 05:59:40.557215 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.557195 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" path="/var/lib/kubelet/pods/45652c65-84b3-4463-92fe-9bb9a8e230f7/volumes" Oct 30 05:59:40.738488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.734831 5072 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 05:59:40.738488 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.734889 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 05:59:40.953011 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.950612 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz"] Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962379 5072 cpu_manager.go:414] "containerMap: removing container" podUID="2423923f-15e4-4204-86de-5a6483ee8356" containerName="maas-api" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962406 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2423923f-15e4-4204-86de-5a6483ee8356" containerName="maas-api" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962429 5072 cpu_manager.go:414] "containerMap: removing container" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962436 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962479 5072 cpu_manager.go:414] "containerMap: removing container" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962485 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962508 5072 cpu_manager.go:414] "containerMap: removing container" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962515 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962720 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2423923f-15e4-4204-86de-5a6483ee8356" containerName="maas-api" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962732 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962746 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="45652c65-84b3-4463-92fe-9bb9a8e230f7" containerName="istio-proxy" Oct 30 05:59:40.965480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.962756 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d474a0-f643-48e6-b1e3-f97070a90298" containerName="istio-proxy" Oct 30 05:59:40.987227 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.987198 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz"] Oct 30 05:59:40.987380 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:40.987324 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.061097 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.061058 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-1b4b7a-openshift-default-dockercfg-w77rm\"" Oct 30 05:59:41.070931 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.070903 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8"] Oct 30 05:59:41.071347 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.071315 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" containerID="cri-o://cd8932f0f5a08da513ca3ced9e939a19b5cd80ddd027de2e178865a8ed60cc86" gracePeriod=30 Oct 30 05:59:41.082188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.082159 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-credential-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.082326 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.082234 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-istio-token\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.082326 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.082262 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-data\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.082451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.082355 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.082451 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.082397 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxbn\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-kube-api-access-kzxbn\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.082586 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.082571 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-certs\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.082675 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.082648 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-envoy\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.082891 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.082867 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9375e831-d43b-41c9-af62-082c05de351e-istio-podinfo\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.082963 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.082940 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9375e831-d43b-41c9-af62-082c05de351e-istiod-ca-cert\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.184645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.184613 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxbn\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-kube-api-access-kzxbn\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.184850 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.184698 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-certs\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.184850 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.184784 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-envoy\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.184975 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.184857 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9375e831-d43b-41c9-af62-082c05de351e-istio-podinfo\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.184975 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.184928 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9375e831-d43b-41c9-af62-082c05de351e-istiod-ca-cert\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.185094 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.185079 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-credential-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.185156 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.185092 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-certs\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.185224 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.185177 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-istio-token\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.185224 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.185211 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-data\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.185343 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.185324 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.185404 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.185385 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-credential-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.185716 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.185689 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9375e831-d43b-41c9-af62-082c05de351e-istiod-ca-cert\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.185830 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.185723 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.185830 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.185767 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-data\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.187180 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.187158 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-envoy\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.187503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.187444 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9375e831-d43b-41c9-af62-082c05de351e-istio-podinfo\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.207910 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.207847 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-istio-token\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.208010 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.207908 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxbn\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-kube-api-access-kzxbn\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.263306 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.263194 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl"] Oct 30 05:59:41.263786 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.263753 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" podUID="2f5ad48b-980d-4740-aa22-2af1faa69ab5" containerName="maas-api" containerID="cri-o://c3b9ef59040784a1671ceb69046ca3f747b4be79451aa093b16f82871f6dba72" gracePeriod=30 Oct 30 05:59:41.308242 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.308211 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:41.782118 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.781554 5072 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 05:59:41.782118 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.781627 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 05:59:41.928515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.921607 5072 generic.go:356] "Generic (PLEG): container finished" podID="2f5ad48b-980d-4740-aa22-2af1faa69ab5" containerID="c3b9ef59040784a1671ceb69046ca3f747b4be79451aa093b16f82871f6dba72" exitCode=0 Oct 30 05:59:41.928515 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:41.921816 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" event={"ID":"2f5ad48b-980d-4740-aa22-2af1faa69ab5","Type":"ContainerDied","Data":"c3b9ef59040784a1671ceb69046ca3f747b4be79451aa093b16f82871f6dba72"} Oct 30 05:59:42.273438 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.272239 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:42.273778 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.273750 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:42.273931 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.273914 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:42.288089 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.288039 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz"] Oct 30 05:59:42.378711 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.378681 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz"] Oct 30 05:59:42.393395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.393377 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.436483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.423690 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-482468-openshift-default-dockercfg-fctb5\"" Oct 30 05:59:42.468477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.455330 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz"] Oct 30 05:59:42.468477 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.465636 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:59:42.488345 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.488314 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-maas-api-tls\") pod \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " Oct 30 05:59:42.488564 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.488528 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-metrics-certs\") pod \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " Oct 30 05:59:42.488657 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.488643 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5ad48b-980d-4740-aa22-2af1faa69ab5-kube-api-access-q2hgv\" (UniqueName: \"kubernetes.io/projected/2f5ad48b-980d-4740-aa22-2af1faa69ab5-kube-api-access-q2hgv\") pod \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " Oct 30 05:59:42.488731 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.488675 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5ad48b-980d-4740-aa22-2af1faa69ab5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5ad48b-980d-4740-aa22-2af1faa69ab5-openshift-service-ca\") pod \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\" (UID: \"2f5ad48b-980d-4740-aa22-2af1faa69ab5\") " Oct 30 05:59:42.488935 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.488910 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-podinfo\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.489016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.488964 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-envoy\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.489016 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.489004 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-certs\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.489110 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.489034 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-data\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.489167 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.489120 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.489220 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.489185 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-credential-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.489416 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.489397 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istiod-ca-cert\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.489967 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.489944 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-token\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.490196 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.490177 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frk2c\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-kube-api-access-frk2c\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.490435 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.489891 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5ad48b-980d-4740-aa22-2af1faa69ab5-openshift-service-ca" pod "2f5ad48b-980d-4740-aa22-2af1faa69ab5" (UID: "2f5ad48b-980d-4740-aa22-2af1faa69ab5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:42.500310 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.500268 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-maas-api-tls" pod "2f5ad48b-980d-4740-aa22-2af1faa69ab5" (UID: "2f5ad48b-980d-4740-aa22-2af1faa69ab5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:42.500618 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.500584 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5ad48b-980d-4740-aa22-2af1faa69ab5-kube-api-access-q2hgv" pod "2f5ad48b-980d-4740-aa22-2af1faa69ab5" (UID: "2f5ad48b-980d-4740-aa22-2af1faa69ab5"). InnerVolumeSpecName "kube-api-access-q2hgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:42.500703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.500632 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-metrics-certs" pod "2f5ad48b-980d-4740-aa22-2af1faa69ab5" (UID: "2f5ad48b-980d-4740-aa22-2af1faa69ab5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.592551 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istiod-ca-cert\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.592602 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-token\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.592689 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frk2c\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-kube-api-access-frk2c\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.592811 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-podinfo\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.592847 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-envoy\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.592879 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-certs\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.592907 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-data\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.592994 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.593055 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-credential-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.593292 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.593309 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f5ad48b-980d-4740-aa22-2af1faa69ab5-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.593324 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2hgv\" (UniqueName: \"kubernetes.io/projected/2f5ad48b-980d-4740-aa22-2af1faa69ab5-kube-api-access-q2hgv\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:42.593551 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.593339 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f5ad48b-980d-4740-aa22-2af1faa69ab5-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:42.594878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.594850 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-certs\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.597544 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.595075 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.597703 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.595202 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-data\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.597815 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.595315 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-credential-socket\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.597919 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.596126 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istiod-ca-cert\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.598017 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.597374 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-envoy\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.598386 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.598350 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-podinfo\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.717499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.688245 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frk2c\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-kube-api-access-frk2c\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.723503 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.721817 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-token\") pod \"e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:42.735993 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.735955 5072 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 05:59:42.736278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.736231 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 05:59:42.782273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:42.781980 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:43.169271 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.169235 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" event={"ID":"9375e831-d43b-41c9-af62-082c05de351e","Type":"ContainerStarted","Data":"3018b74b597349dba960d173f998672bf7f0ca3c1db6a79114fb5fa3c81d7046"} Oct 30 05:59:43.169983 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.169960 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" event={"ID":"9375e831-d43b-41c9-af62-082c05de351e","Type":"ContainerStarted","Data":"dadcc0f5fc74718d1b69c972cc8239b6cedb9e4c557be01ac2ea129a1ebd10a6"} Oct 30 05:59:43.177753 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.173333 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" event={"ID":"2f5ad48b-980d-4740-aa22-2af1faa69ab5","Type":"ContainerDied","Data":"fbfc51e09ba2cada69b5792e0178dc88fe4b7fd5df99f70110188521f9e28c2f"} Oct 30 05:59:43.177753 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.173389 5072 scope.go:122] "RemoveContainer" containerID="c3b9ef59040784a1671ceb69046ca3f747b4be79451aa093b16f82871f6dba72" Oct 30 05:59:43.177753 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.173546 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl" Oct 30 05:59:43.233426 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.233355 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podStartSLOduration=3.233335176 podStartE2EDuration="3.233335176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:43.226804702 +0000 UTC m=+1742.830493990" watchObservedRunningTime="2026-10-30 05:59:43.233335176 +0000 UTC m=+1742.837024437" Oct 30 05:59:43.298668 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.298631 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl"] Oct 30 05:59:43.310759 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.310721 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76b905-686f766f84-zjzpl"] Oct 30 05:59:43.311014 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.310999 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:43.328898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.328862 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 05:59:43.329160 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.329129 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 05:59:43.739041 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.739007 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz"] Oct 30 05:59:43.739182 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.739155 5072 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 05:59:43.739252 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.739198 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 05:59:43.744473 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.744413 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:43.744570 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.744529 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:43.744645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:43.744628 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:44.271019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:44.270964 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" event={"ID":"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c","Type":"ContainerStarted","Data":"5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234"} Oct 30 05:59:44.271889 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:44.271846 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" event={"ID":"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c","Type":"ContainerStarted","Data":"fbc495f341ebae21592d2e21a1ecf3387f4a86dd7fb7652762c0636597c147da"} Oct 30 05:59:44.343817 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:44.343741 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:44.416496 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:44.415987 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podStartSLOduration=2.41596438 podStartE2EDuration="2.41596438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:44.34140256 +0000 UTC m=+1743.945091819" watchObservedRunningTime="2026-10-30 05:59:44.41596438 +0000 UTC m=+1744.019653639" Oct 30 05:59:44.619563 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:44.605197 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5ad48b-980d-4740-aa22-2af1faa69ab5" path="/var/lib/kubelet/pods/2f5ad48b-980d-4740-aa22-2af1faa69ab5/volumes" Oct 30 05:59:44.784080 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:44.784051 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:44.784389 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:44.784368 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:59:44.784586 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:44.784561 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 05:59:45.176950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:45.176917 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:45.178756 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:45.178734 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:45.186839 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:45.186800 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 05:59:45.371483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:45.368541 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:45.392494 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:45.390005 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 05:59:45.799622 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:45.799594 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:46.383938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.373231 5072 generic.go:356] "Generic (PLEG): container finished" podID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerID="cd8932f0f5a08da513ca3ced9e939a19b5cd80ddd027de2e178865a8ed60cc86" exitCode=0 Oct 30 05:59:46.383938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.374350 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" event={"ID":"46689523-9454-44e3-8e76-fb8992bb6c1b","Type":"ContainerDied","Data":"cd8932f0f5a08da513ca3ced9e939a19b5cd80ddd027de2e178865a8ed60cc86"} Oct 30 05:59:46.383938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.374410 5072 scope.go:122] "RemoveContainer" containerID="43193e033ef400484b2378d8dad6253081ff741d413ec465a67e7e765d3b6806" Oct 30 05:59:46.383938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.376391 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:46.383938 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.377830 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 05:59:46.581019 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.580733 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:59:46.707810 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.707708 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-kube-api-access-k4rl6\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-kube-api-access-k4rl6\") pod \"46689523-9454-44e3-8e76-fb8992bb6c1b\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " Oct 30 05:59:46.707982 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.707841 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-token\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-token\") pod \"46689523-9454-44e3-8e76-fb8992bb6c1b\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " Oct 30 05:59:46.707982 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.707890 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-credential-socket\") pod \"46689523-9454-44e3-8e76-fb8992bb6c1b\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " Oct 30 05:59:46.707982 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.707932 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-envoy\") pod \"46689523-9454-44e3-8e76-fb8992bb6c1b\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " Oct 30 05:59:46.708200 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.708041 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-data\") pod \"46689523-9454-44e3-8e76-fb8992bb6c1b\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " Oct 30 05:59:46.708200 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.708073 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-certs\") pod \"46689523-9454-44e3-8e76-fb8992bb6c1b\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " Oct 30 05:59:46.708200 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.708151 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-podinfo\") pod \"46689523-9454-44e3-8e76-fb8992bb6c1b\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " Oct 30 05:59:46.708368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.708213 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-socket\") pod \"46689523-9454-44e3-8e76-fb8992bb6c1b\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " Oct 30 05:59:46.708368 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.708281 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46689523-9454-44e3-8e76-fb8992bb6c1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46689523-9454-44e3-8e76-fb8992bb6c1b-istiod-ca-cert\") pod \"46689523-9454-44e3-8e76-fb8992bb6c1b\" (UID: \"46689523-9454-44e3-8e76-fb8992bb6c1b\") " Oct 30 05:59:46.717151 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.716700 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-certs" pod "46689523-9454-44e3-8e76-fb8992bb6c1b" (UID: "46689523-9454-44e3-8e76-fb8992bb6c1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:46.717151 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.716889 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-socket" pod "46689523-9454-44e3-8e76-fb8992bb6c1b" (UID: "46689523-9454-44e3-8e76-fb8992bb6c1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:46.742757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.740510 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-credential-socket" pod "46689523-9454-44e3-8e76-fb8992bb6c1b" (UID: "46689523-9454-44e3-8e76-fb8992bb6c1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:46.742757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.741740 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-data" pod "46689523-9454-44e3-8e76-fb8992bb6c1b" (UID: "46689523-9454-44e3-8e76-fb8992bb6c1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:46.742757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.742136 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46689523-9454-44e3-8e76-fb8992bb6c1b-istiod-ca-cert" pod "46689523-9454-44e3-8e76-fb8992bb6c1b" (UID: "46689523-9454-44e3-8e76-fb8992bb6c1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:46.742757 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.742380 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-envoy" pod "46689523-9454-44e3-8e76-fb8992bb6c1b" (UID: "46689523-9454-44e3-8e76-fb8992bb6c1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:46.743100 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.742860 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-token" pod "46689523-9454-44e3-8e76-fb8992bb6c1b" (UID: "46689523-9454-44e3-8e76-fb8992bb6c1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:46.743100 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.743016 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-podinfo" pod "46689523-9454-44e3-8e76-fb8992bb6c1b" (UID: "46689523-9454-44e3-8e76-fb8992bb6c1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:46.744908 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.744880 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-kube-api-access-k4rl6" pod "46689523-9454-44e3-8e76-fb8992bb6c1b" (UID: "46689523-9454-44e3-8e76-fb8992bb6c1b"). InnerVolumeSpecName "kube-api-access-k4rl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:46.832483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.830721 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:46.832483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.830756 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:46.832483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.830772 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:46.832483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.830785 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:46.832483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.830799 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46689523-9454-44e3-8e76-fb8992bb6c1b-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:46.832483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.830813 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4rl6\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-kube-api-access-k4rl6\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:46.832483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.830829 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:46.832483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.830842 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:46.832483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:46.830855 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46689523-9454-44e3-8e76-fb8992bb6c1b-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:47.385818 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.385777 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" event={"ID":"46689523-9454-44e3-8e76-fb8992bb6c1b","Type":"ContainerDied","Data":"b07125552c2e0762529e4342f21da3cab86984be12c33e4b4e03ce7345af90f5"} Oct 30 05:59:47.386314 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.385989 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8" Oct 30 05:59:47.386507 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.386290 5072 scope.go:122] "RemoveContainer" containerID="cd8932f0f5a08da513ca3ced9e939a19b5cd80ddd027de2e178865a8ed60cc86" Oct 30 05:59:47.471088 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.471049 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8"] Oct 30 05:59:47.476873 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.476842 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-530a5e-openshift-default-8f6ff7df-9gmp8"] Oct 30 05:59:47.957149 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.957106 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft"] Oct 30 05:59:47.958282 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.958262 5072 cpu_manager.go:414] "containerMap: removing container" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" Oct 30 05:59:47.958376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.958284 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" Oct 30 05:59:47.958376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.958304 5072 cpu_manager.go:414] "containerMap: removing container" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" Oct 30 05:59:47.958376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.958311 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" Oct 30 05:59:47.958376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.958355 5072 cpu_manager.go:414] "containerMap: removing container" podUID="2f5ad48b-980d-4740-aa22-2af1faa69ab5" containerName="maas-api" Oct 30 05:59:47.958376 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.958364 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5ad48b-980d-4740-aa22-2af1faa69ab5" containerName="maas-api" Oct 30 05:59:47.958586 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.958571 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" Oct 30 05:59:47.958625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.958592 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" containerName="istio-proxy" Oct 30 05:59:47.958625 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.958606 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5ad48b-980d-4740-aa22-2af1faa69ab5" containerName="maas-api" Oct 30 05:59:47.965572 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.965546 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 05:59:47.981898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.981868 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-1b4b7a\"" Oct 30 05:59:47.981898 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.981866 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-1b4b7a-dockercfg-qqh5l\"" Oct 30 05:59:47.986237 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:47.986206 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft"] Oct 30 05:59:48.064853 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.063598 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6pd\" (UniqueName: \"kubernetes.io/projected/51a6043d-0206-452f-92a8-3a6324d213aa-kube-api-access-sh6pd\") pod \"payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft\" (UID: \"51a6043d-0206-452f-92a8-3a6324d213aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 05:59:48.064853 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.063736 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a6043d-0206-452f-92a8-3a6324d213aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft\" (UID: \"51a6043d-0206-452f-92a8-3a6324d213aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 05:59:48.198481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.193151 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6pd\" (UniqueName: \"kubernetes.io/projected/51a6043d-0206-452f-92a8-3a6324d213aa-kube-api-access-sh6pd\") pod \"payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft\" (UID: \"51a6043d-0206-452f-92a8-3a6324d213aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 05:59:48.198481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.193337 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a6043d-0206-452f-92a8-3a6324d213aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft\" (UID: \"51a6043d-0206-452f-92a8-3a6324d213aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 05:59:48.198481 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.194738 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a6043d-0206-452f-92a8-3a6324d213aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft\" (UID: \"51a6043d-0206-452f-92a8-3a6324d213aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 05:59:48.325535 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.324247 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6pd\" (UniqueName: \"kubernetes.io/projected/51a6043d-0206-452f-92a8-3a6324d213aa-kube-api-access-sh6pd\") pod \"payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft\" (UID: \"51a6043d-0206-452f-92a8-3a6324d213aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 05:59:48.587032 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.586753 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 05:59:48.603782 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.603750 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46689523-9454-44e3-8e76-fb8992bb6c1b" path="/var/lib/kubelet/pods/46689523-9454-44e3-8e76-fb8992bb6c1b/volumes" Oct 30 05:59:48.795590 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.795545 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8"] Oct 30 05:59:48.797562 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:48.796433 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" containerID="cri-o://321bde7f2a923dcf50795b1a6b91ef7c0e0cc5720d48fdca0a3c357cbff3d1e8" gracePeriod=30 Oct 30 05:59:49.880124 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:49.875223 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft"] Oct 30 05:59:50.554490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.533544 5072 patch_prober.go:29] interesting pod/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:59:50.554490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.533609 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:59:50.554490 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.533883 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx"] Oct 30 05:59:50.748524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.748415 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.748524 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.748499 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istiod-ca-cert\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.748789 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.748591 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-podinfo\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.748789 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.748678 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-token\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.748789 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.748743 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-certs\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.748977 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.748827 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-credential-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.748977 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.748932 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-data\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.748977 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.748972 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2pl\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-kube-api-access-zj2pl\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.749149 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.749040 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-envoy\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.752554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.751845 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx"] Oct 30 05:59:50.767395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.767369 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:50.792400 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.784234 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1dbaf6-openshift-default-dockercfg-vtk9c\"" Oct 30 05:59:50.882248 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:50.882198 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-token\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.035635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.032946 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-certs\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.035635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.033053 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-credential-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.035635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.033192 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-data\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.035635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.033238 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2pl\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-kube-api-access-zj2pl\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.035635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.033319 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-envoy\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.035635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.033406 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.062499 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.061903 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istiod-ca-cert\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.063561 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.062768 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-podinfo\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.067026 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.066996 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-certs\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.067185 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.067167 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istiod-ca-cert\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.067514 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.067276 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-credential-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.067628 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.067541 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-data\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.093243 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.093210 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.094624 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.094601 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-envoy\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.110889 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.109154 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2pl\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-kube-api-access-zj2pl\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.117308 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.117268 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-token\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.186961 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.186848 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-podinfo\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.187528 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.187501 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:51.189549 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.189098 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" event={"ID":"51a6043d-0206-452f-92a8-3a6324d213aa","Type":"ContainerStarted","Data":"83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e"} Oct 30 05:59:51.189549 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.189191 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" event={"ID":"51a6043d-0206-452f-92a8-3a6324d213aa","Type":"ContainerStarted","Data":"36c74ffad0d9cb7e8d28f747b6d9fec94196f95829e23cd26a56be11a29948e9"} Oct 30 05:59:51.190635 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.190501 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 05:59:51.255485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:51.242679 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" podStartSLOduration=4.242656369 podStartE2EDuration="4.242656369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:51.237775401 +0000 UTC m=+1750.841464662" watchObservedRunningTime="2026-10-30 05:59:51.242656369 +0000 UTC m=+1750.846345627" Oct 30 05:59:52.089613 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:52.089514 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx"] Oct 30 05:59:52.200407 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:59:52.200370 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9f028f_843f_4b0b_bdb4_01b3f4b98a9a.slice/crio-e8f7f62cfd2b6651b1186bc296f4472bbc2fab19263c971e6520e63a59f52772 WatchSource:0}: Error finding container e8f7f62cfd2b6651b1186bc296f4472bbc2fab19263c971e6520e63a59f52772: Status 404 returned error can't find the container with id e8f7f62cfd2b6651b1186bc296f4472bbc2fab19263c971e6520e63a59f52772 Oct 30 05:59:52.203886 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:52.203850 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:52.203992 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:52.203922 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:52.203992 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:52.203953 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:52.974970 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:52.973980 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f"] Oct 30 05:59:52.988480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:52.987762 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f"] Oct 30 05:59:52.988480 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:52.987916 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.095383 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.095354 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f28e2d-openshift-default-dockercfg-jnngq\"" Oct 30 05:59:53.100286 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.100255 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-envoy\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.100552 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.100532 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e614f11-3285-4826-aec4-ffa24b359348-istiod-ca-cert\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.100676 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.100661 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspmh\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-kube-api-access-fspmh\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.100799 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.100786 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-certs\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.100911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.100896 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-data\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.101049 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.101035 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-istio-token\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.101161 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.101148 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-socket\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.101278 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.101265 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e614f11-3285-4826-aec4-ffa24b359348-istio-podinfo\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.101409 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.101395 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-credential-socket\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.202905 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.202862 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e614f11-3285-4826-aec4-ffa24b359348-istiod-ca-cert\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.203099 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.202954 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspmh\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-kube-api-access-fspmh\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.203099 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.202996 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-certs\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.203099 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.203031 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-data\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.203256 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.203107 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-istio-token\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.203256 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.203155 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-socket\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.203256 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.203227 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e614f11-3285-4826-aec4-ffa24b359348-istio-podinfo\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.203408 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.203277 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-credential-socket\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.203531 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.203513 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-envoy\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.205395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.203605 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-certs\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.205395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.204242 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-socket\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.205395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.204412 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-data\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.205395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.204537 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-credential-socket\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.205395 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.205310 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e614f11-3285-4826-aec4-ffa24b359348-istiod-ca-cert\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.207251 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.207227 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-envoy\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.207708 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.207687 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e614f11-3285-4826-aec4-ffa24b359348-istio-podinfo\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.253540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.252871 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" event={"ID":"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a","Type":"ContainerStarted","Data":"c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03"} Oct 30 05:59:53.253540 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.252930 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" event={"ID":"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a","Type":"ContainerStarted","Data":"e8f7f62cfd2b6651b1186bc296f4472bbc2fab19263c971e6520e63a59f52772"} Oct 30 05:59:53.279554 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.276584 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-istio-token\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.291188 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.291157 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspmh\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-kube-api-access-fspmh\") pod \"e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.352820 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.352578 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:53.445311 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:53.445242 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podStartSLOduration=3.445222382 podStartE2EDuration="3.445222382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:53.315659969 +0000 UTC m=+1752.919349273" watchObservedRunningTime="2026-10-30 05:59:53.445222382 +0000 UTC m=+1753.048911641" Oct 30 05:59:53.996718 ip-10-0-138-200 kubenswrapper[5072]: E1030 05:59:53.990955 5072 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee624f7_b2c8_48e1_b001_f2d58bc7a905.slice/crio-321bde7f2a923dcf50795b1a6b91ef7c0e0cc5720d48fdca0a3c357cbff3d1e8.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:59:54.069827 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.069744 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59"] Oct 30 05:59:54.169099 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.161016 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.286591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.265355 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-765de9-openshift-default-dockercfg-72jrs\"" Oct 30 05:59:54.289273 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.289247 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:54.289530 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.289511 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59"] Oct 30 05:59:54.374591 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.353041 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:54.685103 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.685069 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cfb151-b682-400e-a174-f37d2c65f08a-istiod-ca-cert\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.685367 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.685349 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.685523 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.685507 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbfcf\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-kube-api-access-bbfcf\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.685685 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.685668 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-envoy\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.685924 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.685898 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-data\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.686125 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.686109 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cfb151-b682-400e-a174-f37d2c65f08a-istio-podinfo\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.686253 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.686241 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-istio-token\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.686399 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.686386 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-credential-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.686546 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.686528 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-certs\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.773079 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.773032 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:54.773348 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.773327 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:54.783020 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.773472 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:54.784126 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.773774 5072 generic.go:356] "Generic (PLEG): container finished" podID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerID="321bde7f2a923dcf50795b1a6b91ef7c0e0cc5720d48fdca0a3c357cbff3d1e8" exitCode=0 Oct 30 05:59:54.799475 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.799437 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:54.799595 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.799561 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 05:59:54.799645 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.799627 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" event={"ID":"8ee624f7-b2c8-48e1-b001-f2d58bc7a905","Type":"ContainerDied","Data":"321bde7f2a923dcf50795b1a6b91ef7c0e0cc5720d48fdca0a3c357cbff3d1e8"} Oct 30 05:59:54.799684 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.799675 5072 scope.go:122] "RemoveContainer" containerID="b21b202a79fcb11e93b95f25d6f654390b31e35ee448e4cf717622ecb141b845" Oct 30 05:59:54.801526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.801504 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f"] Oct 30 05:59:54.804911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.804597 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cfb151-b682-400e-a174-f37d2c65f08a-istio-podinfo\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.804911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.804658 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-istio-token\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.804911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.804748 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-credential-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.804911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.804784 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-certs\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.804911 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.804866 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cfb151-b682-400e-a174-f37d2c65f08a-istiod-ca-cert\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.805136 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.804923 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.805136 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.804954 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbfcf\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-kube-api-access-bbfcf\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.805136 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.804997 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-envoy\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.805233 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.805187 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-data\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.808239 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.808214 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-certs\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.808878 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.808855 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.809205 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.808911 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cfb151-b682-400e-a174-f37d2c65f08a-istiod-ca-cert\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.809292 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.809137 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-credential-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.809371 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.809166 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-data\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.860786 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.857765 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-envoy\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.875373 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.875265 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-istio-token\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.880478 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.880405 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cfb151-b682-400e-a174-f37d2c65f08a-istio-podinfo\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.902836 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.902790 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbfcf\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-kube-api-access-bbfcf\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.959919 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.959807 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:54.991131 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:54.990774 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.008587 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-kube-api-access-gqcwc\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-kube-api-access-gqcwc\") pod \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.008648 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-token\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-token\") pod \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.008678 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-envoy\") pod \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.008716 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-certs\") pod \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.008770 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istiod-ca-cert\") pod \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.008825 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-socket\") pod \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.008875 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-credential-socket\") pod \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.008949 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-podinfo\") pod \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.009026 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-data\") pod \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\" (UID: \"8ee624f7-b2c8-48e1-b001-f2d58bc7a905\") " Oct 30 05:59:55.024526 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.016743 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-certs" pod "8ee624f7-b2c8-48e1-b001-f2d58bc7a905" (UID: "8ee624f7-b2c8-48e1-b001-f2d58bc7a905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:55.033648 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.032801 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istiod-ca-cert" pod "8ee624f7-b2c8-48e1-b001-f2d58bc7a905" (UID: "8ee624f7-b2c8-48e1-b001-f2d58bc7a905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:55.033648 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.033052 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-socket" pod "8ee624f7-b2c8-48e1-b001-f2d58bc7a905" (UID: "8ee624f7-b2c8-48e1-b001-f2d58bc7a905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:55.033648 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.033213 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-credential-socket" pod "8ee624f7-b2c8-48e1-b001-f2d58bc7a905" (UID: "8ee624f7-b2c8-48e1-b001-f2d58bc7a905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:55.035553 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.033993 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-data" pod "8ee624f7-b2c8-48e1-b001-f2d58bc7a905" (UID: "8ee624f7-b2c8-48e1-b001-f2d58bc7a905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:55.065720 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.065651 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-token" pod "8ee624f7-b2c8-48e1-b001-f2d58bc7a905" (UID: "8ee624f7-b2c8-48e1-b001-f2d58bc7a905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:55.068704 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.068666 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-envoy" pod "8ee624f7-b2c8-48e1-b001-f2d58bc7a905" (UID: "8ee624f7-b2c8-48e1-b001-f2d58bc7a905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:55.070879 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.070785 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-kube-api-access-gqcwc" pod "8ee624f7-b2c8-48e1-b001-f2d58bc7a905" (UID: "8ee624f7-b2c8-48e1-b001-f2d58bc7a905"). InnerVolumeSpecName "kube-api-access-gqcwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:55.072102 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.071992 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-podinfo" pod "8ee624f7-b2c8-48e1-b001-f2d58bc7a905" (UID: "8ee624f7-b2c8-48e1-b001-f2d58bc7a905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:55.147436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.146627 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:55.147436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.146670 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqcwc\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-kube-api-access-gqcwc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:55.147436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.146686 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:55.147436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.146702 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:55.147436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.146716 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:55.147436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.146728 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:55.147436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.146740 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:55.147436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.146755 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:55.147436 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.146769 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee624f7-b2c8-48e1-b001-f2d58bc7a905-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 05:59:55.445483 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.429965 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59"] Oct 30 05:59:55.455715 ip-10-0-138-200 kubenswrapper[5072]: W1030 05:59:55.455101 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cfb151_b682_400e_a174_f37d2c65f08a.slice/crio-4ac7b73df69248cc742ea2d7e11135c7abcb7c0c345f7bc2c93b1150e128da8a WatchSource:0}: Error finding container 4ac7b73df69248cc742ea2d7e11135c7abcb7c0c345f7bc2c93b1150e128da8a: Status 404 returned error can't find the container with id 4ac7b73df69248cc742ea2d7e11135c7abcb7c0c345f7bc2c93b1150e128da8a Oct 30 05:59:55.458265 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.458228 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:55.458370 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.458302 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:55.458370 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.458334 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 05:59:55.796585 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.796498 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" event={"ID":"45cfb151-b682-400e-a174-f37d2c65f08a","Type":"ContainerStarted","Data":"f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad"} Oct 30 05:59:55.796719 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.796588 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" event={"ID":"45cfb151-b682-400e-a174-f37d2c65f08a","Type":"ContainerStarted","Data":"4ac7b73df69248cc742ea2d7e11135c7abcb7c0c345f7bc2c93b1150e128da8a"} Oct 30 05:59:55.798355 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.798331 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" Oct 30 05:59:55.798539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.798381 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8" event={"ID":"8ee624f7-b2c8-48e1-b001-f2d58bc7a905","Type":"ContainerDied","Data":"54608958c81feac0d57c77b5f22747db2bdbde73b7f5672987bf73a0049afbdd"} Oct 30 05:59:55.798539 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.798438 5072 scope.go:122] "RemoveContainer" containerID="321bde7f2a923dcf50795b1a6b91ef7c0e0cc5720d48fdca0a3c357cbff3d1e8" Oct 30 05:59:55.802226 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.801527 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" event={"ID":"0e614f11-3285-4826-aec4-ffa24b359348","Type":"ContainerStarted","Data":"15e9fb0be3f0f9847ff14b84421814090d645fcacdbca5206c521b23db44e68f"} Oct 30 05:59:55.802226 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.801604 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" event={"ID":"0e614f11-3285-4826-aec4-ffa24b359348","Type":"ContainerStarted","Data":"f0ba67e1312dfa1ff460323c73c28b2b153b8e8e1fe97bcb799852bd8fc888ef"} Oct 30 05:59:55.853082 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.852951 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podStartSLOduration=2.852915897 podStartE2EDuration="2.852915897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:55.84074256 +0000 UTC m=+1755.444431839" watchObservedRunningTime="2026-10-30 05:59:55.852915897 +0000 UTC m=+1755.456605155" Oct 30 05:59:55.929136 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.929078 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podStartSLOduration=3.929062345 podStartE2EDuration="3.929062345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:55.925620883 +0000 UTC m=+1755.529310153" watchObservedRunningTime="2026-10-30 05:59:55.929062345 +0000 UTC m=+1755.532751601" Oct 30 05:59:55.960846 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.960811 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:55.979342 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.979305 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8"] Oct 30 05:59:55.990074 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:55.990048 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4323f6-openshift-default-69dcfccf67-k7cl8"] Oct 30 05:59:56.452485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:56.365195 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:56.452485 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:56.437771 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:56.545950 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:56.545890 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" path="/var/lib/kubelet/pods/8ee624f7-b2c8-48e1-b001-f2d58bc7a905/volumes" Oct 30 05:59:56.816638 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:56.816533 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:56.818039 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:56.817989 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 05:59:56.969783 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:56.969750 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:57.846350 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:57.843088 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 05:59:57.846350 ip-10-0-138-200 kubenswrapper[5072]: I1030 05:59:57.844240 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 06:00:00.269541 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.267127 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx"] Oct 30 06:00:00.269541 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.268007 5072 cpu_manager.go:414] "containerMap: removing container" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" Oct 30 06:00:00.269541 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.268022 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" Oct 30 06:00:00.269541 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.268053 5072 cpu_manager.go:414] "containerMap: removing container" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" Oct 30 06:00:00.269541 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.268059 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" Oct 30 06:00:00.269541 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.268206 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" Oct 30 06:00:00.272203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.272168 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.389865 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.389830 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx"] Oct 30 06:00:00.429523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.427191 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.429523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.427252 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.429523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.427299 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f17885-3892-490b-bc8b-8499db744394-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.429523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.427359 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.429523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.427404 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.429523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.427481 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whts4\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-kube-api-access-whts4\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.429523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.427540 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.429523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.427580 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.429523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.427595 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f17885-3892-490b-bc8b-8499db744394-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.529384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529290 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.529384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529361 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f17885-3892-490b-bc8b-8499db744394-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.529804 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529494 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.529804 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529572 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.529804 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529656 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whts4\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-kube-api-access-whts4\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.529804 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529761 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.530010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529803 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.530010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529827 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.530010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529853 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f17885-3892-490b-bc8b-8499db744394-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.530010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.529892 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.530208 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.530186 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.530274 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.530235 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.530505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.530451 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.531088 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.531061 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f17885-3892-490b-bc8b-8499db744394-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.532629 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.532603 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.532851 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.532833 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f17885-3892-490b-bc8b-8499db744394-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.556975 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.556943 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whts4\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-kube-api-access-whts4\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.559395 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.559361 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.595138 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.595098 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:00.743175 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:00:00.743089 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:00:00.875316 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.875280 5072 generic.go:356] "Generic (PLEG): container finished" podID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerID="6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48" exitCode=0 Oct 30 06:00:00.875599 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.875540 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" event={"ID":"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72","Type":"ContainerDied","Data":"6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48"} Oct 30 06:00:00.876063 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.876043 5072 scope.go:122] "RemoveContainer" containerID="6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48" Oct 30 06:00:00.879979 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.879933 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:00.879979 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.880137 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:00.880442 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.880271 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:00.941583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.941542 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx"] Oct 30 06:00:00.979731 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:00:00.965211 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f17885_3892_490b_bc8b_8499db744394.slice/crio-d6604e6bccc9d2821a7c4c429d5170ad8e1ab52a0f1d2934c467ffae6e3be9e1 WatchSource:0}: Error finding container d6604e6bccc9d2821a7c4c429d5170ad8e1ab52a0f1d2934c467ffae6e3be9e1: Status 404 returned error can't find the container with id d6604e6bccc9d2821a7c4c429d5170ad8e1ab52a0f1d2934c467ffae6e3be9e1 Oct 30 06:00:00.980102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.980057 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:00.980246 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.980157 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:00.980246 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:00.980209 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:01.886762 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:01.886725 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" event={"ID":"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72","Type":"ContainerStarted","Data":"cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d"} Oct 30 06:00:01.889243 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:01.889222 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" event={"ID":"d7f17885-3892-490b-bc8b-8499db744394","Type":"ContainerStarted","Data":"a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14"} Oct 30 06:00:01.889330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:01.889263 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" event={"ID":"d7f17885-3892-490b-bc8b-8499db744394","Type":"ContainerStarted","Data":"d6604e6bccc9d2821a7c4c429d5170ad8e1ab52a0f1d2934c467ffae6e3be9e1"} Oct 30 06:00:01.985450 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:01.985395 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podStartSLOduration=1.985381971 podStartE2EDuration="1.985381971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:01.979692078 +0000 UTC m=+1761.583381338" watchObservedRunningTime="2026-10-30 06:00:01.985381971 +0000 UTC m=+1761.589071228" Oct 30 06:00:02.284839 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.284724 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 06:00:02.286304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.286275 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:00:02.286521 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.286335 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 06:00:02.595653 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.595572 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:02.597432 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.597408 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 06:00:02.597549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.597453 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 06:00:02.764484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.764361 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 06:00:02.904674 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.904643 5072 generic.go:356] "Generic (PLEG): container finished" podID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerID="e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0" exitCode=0 Oct 30 06:00:02.905102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.904992 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" event={"ID":"fab0bf3a-d180-4e27-a171-33b97bec9497","Type":"ContainerDied","Data":"e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0"} Oct 30 06:00:02.905350 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.905332 5072 scope.go:122] "RemoveContainer" containerID="e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0" Oct 30 06:00:02.907553 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.907523 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:02.907650 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.907596 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:02.907650 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:02.907635 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:03.282531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:03.278712 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 06:00:03.292490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:03.289216 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:00:03.292490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:03.289274 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 06:00:03.602545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:03.600161 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 06:00:03.602545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:03.600228 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 06:00:04.243508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:04.222219 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" event={"ID":"fab0bf3a-d180-4e27-a171-33b97bec9497","Type":"ContainerStarted","Data":"ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302"} Oct 30 06:00:04.331485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:04.329965 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:00:04.331485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:04.330027 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 06:00:04.385286 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:04.385255 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 06:00:04.439644 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:04.439610 5072 patch_prober.go:29] interesting pod/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:00:04.440370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:04.440331 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 06:00:04.618005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:04.617964 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 06:00:04.618347 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:04.618307 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 06:00:04.736484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:04.715014 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Oct 30 06:00:05.307482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:05.306904 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:00:05.307482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:05.306977 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 06:00:05.400486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:05.390866 5072 patch_prober.go:29] interesting pod/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:00:05.400486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:05.390931 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 06:00:05.595962 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:05.595868 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 06:00:05.596258 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:05.596225 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 06:00:06.294484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:06.292501 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:00:06.294484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:06.292606 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 06:00:06.393485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:06.386114 5072 patch_prober.go:29] interesting pod/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:00:06.393485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:06.386169 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 06:00:06.608551 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:06.605037 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 06:00:06.608551 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:06.605105 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 06:00:06.813489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:06.813430 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 06:00:06.824588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:06.821097 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:06.977538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:06.966931 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.297425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.288897 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:00:07.297425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.288963 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 06:00:07.386491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.385801 5072 patch_prober.go:29] interesting pod/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:00:07.386491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.385864 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 06:00:07.600501 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.600387 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 06:00:07.626483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.601058 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 06:00:07.729504 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.660989 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.998711 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.944993 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz"] Oct 30 06:00:07.998711 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.945303 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.998711 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.970353 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:08.091484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:07.999214 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee624f7-b2c8-48e1-b001-f2d58bc7a905" containerName="istio-proxy" Oct 30 06:00:08.274169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.262314 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Oct 30 06:00:08.278696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.278651 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:08.282739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.282707 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:00:08.282998 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.282982 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:00:08.283127 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.283115 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz"] Oct 30 06:00:08.284094 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.284075 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.284621 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.284603 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:00:08.287069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.287030 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:00:08.287180 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.287095 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 06:00:08.319748 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.310195 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8de592-openshift-default-dockercfg-kzz2m\"" Oct 30 06:00:08.390481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.389961 5072 patch_prober.go:29] interesting pod/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:00:08.390481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.390029 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 06:00:08.485680 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.485641 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjl45\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-kube-api-access-hjl45\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.485967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.485944 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-data\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.486103 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.486088 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.486311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.486293 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-token\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.486499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.486481 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istiod-ca-cert\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.486722 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.486703 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-certs\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.486870 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.486855 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-credential-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.487043 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.487028 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-envoy\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.487196 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.487180 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-podinfo\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.589585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.589487 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjl45\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-kube-api-access-hjl45\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.589882 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.589862 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-data\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.590018 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.590002 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.590213 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.590197 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-token\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.592564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.592541 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istiod-ca-cert\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.593390 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.593364 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-certs\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.593591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.593574 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-credential-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.593789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.593771 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-envoy\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.594555 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.594530 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-podinfo\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.622099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.603782 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 06:00:08.622099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.603839 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 06:00:08.651773 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.643162 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-podinfo\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.654252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.654200 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istiod-ca-cert\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.654778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.654756 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-certs\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.655324 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.655303 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-credential-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.730673 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.728843 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-data\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.730673 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.729097 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.730673 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.729296 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-envoy\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.885496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.885278 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-token\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:08.941684 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:08.938853 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:09.046844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.045955 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:00:09.072498 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.063561 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjl45\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-kube-api-access-hjl45\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:09.180776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.180680 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:09.200936 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.200901 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:09.342828 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.342773 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 06:00:09.394513 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.385663 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 06:00:09.494176 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.494081 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:00:09.793582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.791105 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 06:00:09.793582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.730552 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 06:00:09.793582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.622618 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 06:00:09.793582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.650695 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 06:00:09.793582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:09.792486 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 06:00:10.099674 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.099640 5072 patch_prober.go:29] interesting pod/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:00:10.099930 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.099894 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 06:00:10.161112 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.161078 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 06:00:10.287412 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.286784 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:00:10.287412 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.286846 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 06:00:10.387523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.384605 5072 patch_prober.go:29] interesting pod/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:00:10.387523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.384667 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 06:00:10.469142 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.469065 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz"] Oct 30 06:00:10.483717 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.480986 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:10.483985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.483955 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:10.484138 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.484119 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:10.606216 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.606186 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:10.607420 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.607364 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:10.639569 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:10.639452 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:00:11.055131 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:11.055036 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" event={"ID":"52f96c85-f7ff-4335-85e9-2a998e9b8b66","Type":"ContainerStarted","Data":"3f785a16d41b9b6ce314f1ae38ee73256bff6f66e43350b7874df874d55f09f5"} Oct 30 06:00:11.055131 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:11.055093 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" event={"ID":"52f96c85-f7ff-4335-85e9-2a998e9b8b66","Type":"ContainerStarted","Data":"25b244d4a7d6d97d64ca2870a9365027d3026c2d3693155adfe5c5c46ef868ce"} Oct 30 06:00:11.099636 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:11.099583 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" podStartSLOduration=4.099568308 podStartE2EDuration="4.099568308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:11.096738074 +0000 UTC m=+1770.700427332" watchObservedRunningTime="2026-10-30 06:00:11.099568308 +0000 UTC m=+1770.703257561" Oct 30 06:00:11.201824 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:11.201789 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:11.218214 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:11.218184 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:11.306537 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:11.304059 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 06:00:11.335482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:11.328325 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 06:00:11.384886 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:11.384841 5072 patch_prober.go:29] interesting pod/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:00:11.385082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:11.384901 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 06:00:12.063519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:12.063448 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:12.064410 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:12.064391 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:00:12.385263 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:12.385218 5072 patch_prober.go:29] interesting pod/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:00:12.385502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:12.385286 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 06:00:13.389436 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:13.389399 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 06:00:13.390541 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:13.390519 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 06:00:25.805877 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:25.805836 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc"] Oct 30 06:00:25.842961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:25.842532 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc"] Oct 30 06:00:25.842961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:25.842655 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:25.856655 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:25.856629 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-1b4b7a-serving-cert\"" Oct 30 06:00:25.879690 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:25.879649 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5fl2\" (UniqueName: \"kubernetes.io/projected/7ec25d1d-6e7e-4142-89c6-9a2391877e82-kube-api-access-x5fl2\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:25.879905 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:25.879847 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:25.880769 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:25.880738 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:25.880891 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:25.880799 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec25d1d-6e7e-4142-89c6-9a2391877e82-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.017314 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.017274 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.017864 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.017837 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.018045 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.018025 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec25d1d-6e7e-4142-89c6-9a2391877e82-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.026449 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.026401 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5fl2\" (UniqueName: \"kubernetes.io/projected/7ec25d1d-6e7e-4142-89c6-9a2391877e82-kube-api-access-x5fl2\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.028700 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.028660 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec25d1d-6e7e-4142-89c6-9a2391877e82-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.063161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.062082 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.067211 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.067164 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.071196 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.071165 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5fl2\" (UniqueName: \"kubernetes.io/projected/7ec25d1d-6e7e-4142-89c6-9a2391877e82-kube-api-access-x5fl2\") pod \"maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.162033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.161990 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:26.487756 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:26.487724 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc"] Oct 30 06:00:26.487905 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:00:26.487819 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec25d1d_6e7e_4142_89c6_9a2391877e82.slice/crio-626e7ec0fc021c3cc1a580c70fe67e38e30bc59cd99bf72f467c3f173eef1a60 WatchSource:0}: Error finding container 626e7ec0fc021c3cc1a580c70fe67e38e30bc59cd99bf72f467c3f173eef1a60: Status 404 returned error can't find the container with id 626e7ec0fc021c3cc1a580c70fe67e38e30bc59cd99bf72f467c3f173eef1a60 Oct 30 06:00:27.457125 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.457079 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" event={"ID":"7ec25d1d-6e7e-4142-89c6-9a2391877e82","Type":"ContainerStarted","Data":"1240e3948d5a672917ce94d8bcb1e926a0a86e224104222aa2ef285dfabe838b"} Oct 30 06:00:27.457681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.457173 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:27.457681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.457192 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" event={"ID":"7ec25d1d-6e7e-4142-89c6-9a2391877e82","Type":"ContainerStarted","Data":"626e7ec0fc021c3cc1a580c70fe67e38e30bc59cd99bf72f467c3f173eef1a60"} Oct 30 06:00:27.499195 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.499136 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" podStartSLOduration=2.064192097 podStartE2EDuration="2.499122553s" totalImagesPullingTime="434.930456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:27.495829296 +0000 UTC m=+1787.099518554" watchObservedRunningTime="2026-10-30 06:00:27.499122553 +0000 UTC m=+1787.102811809" Oct 30 06:00:27.805672 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.805590 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74"] Oct 30 06:00:27.809381 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.809356 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:27.824614 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.824584 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-482468-serving-cert\"" Oct 30 06:00:27.843842 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.843805 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74"] Oct 30 06:00:27.970484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.970431 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-metrics-certs\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:27.970688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.970542 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4tc6\" (UniqueName: \"kubernetes.io/projected/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-kube-api-access-m4tc6\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:27.970688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.970589 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:27.970688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:27.970659 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-maas-api-tls\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.083531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.076854 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-metrics-certs\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.083531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.076971 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4tc6\" (UniqueName: \"kubernetes.io/projected/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-kube-api-access-m4tc6\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.083531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.077029 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.083531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.077147 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-maas-api-tls\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.087362 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.087326 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-metrics-certs\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.088225 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.088192 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.110547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.109660 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4tc6\" (UniqueName: \"kubernetes.io/projected/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-kube-api-access-m4tc6\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.124932 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.124900 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-maas-api-tls\") pod \"maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.140824 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.140788 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:28.494199 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:00:28.494159 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a96066_f1b8_4c2b_964d_54fbf3f7f41a.slice/crio-334508b55cdcce0bb3f1c4e76b6407e060b5b2773d07e01339219f4b20889e30 WatchSource:0}: Error finding container 334508b55cdcce0bb3f1c4e76b6407e060b5b2773d07e01339219f4b20889e30: Status 404 returned error can't find the container with id 334508b55cdcce0bb3f1c4e76b6407e060b5b2773d07e01339219f4b20889e30 Oct 30 06:00:28.495048 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:28.495008 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74"] Oct 30 06:00:29.475737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:29.475703 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" event={"ID":"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a","Type":"ContainerStarted","Data":"f92cad136c4a36e6d92312fde07d0dd3fb052f93ab4be36e8729f4c33b14f0d3"} Oct 30 06:00:29.475933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:29.475759 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" event={"ID":"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a","Type":"ContainerStarted","Data":"334508b55cdcce0bb3f1c4e76b6407e060b5b2773d07e01339219f4b20889e30"} Oct 30 06:00:29.475933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:29.475791 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:29.518771 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:29.518715 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" podStartSLOduration=2.187967145 podStartE2EDuration="2.518701098s" totalImagesPullingTime="330.733953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:29.514734533 +0000 UTC m=+1789.118423790" watchObservedRunningTime="2026-10-30 06:00:29.518701098 +0000 UTC m=+1789.122390355" Oct 30 06:00:30.404022 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.403986 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5"] Oct 30 06:00:30.409596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.409569 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.461139 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.461100 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5"] Oct 30 06:00:30.568521 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.568481 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdnm\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-kube-api-access-5jdnm\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.569043 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.568799 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.569043 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.568933 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.569184 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.569107 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.569237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.569184 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.569237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.569226 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.569573 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.569533 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.569695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.569622 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.569695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.569681 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.670846 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.670754 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.670846 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.670806 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671083 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.670885 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdnm\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-kube-api-access-5jdnm\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671083 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671006 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671199 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671107 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671199 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671111 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671199 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671157 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671199 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671180 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671205 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671240 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671324 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671613 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671531 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671716 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.671818 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.671747 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.673446 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.673419 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.673893 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.673870 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.692913 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.692882 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.693222 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.693200 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdnm\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-kube-api-access-5jdnm\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:30.713412 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:00:30.713376 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:00:30.733119 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:30.733092 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:31.058668 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:00:31.058637 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715162c8_0a95_4c0d_b15e_3e57c3b707f3.slice/crio-a2bc7afdc85a97428028c2548b534d7d6d2c9f043dc74309f001e58480e16e56 WatchSource:0}: Error finding container a2bc7afdc85a97428028c2548b534d7d6d2c9f043dc74309f001e58480e16e56: Status 404 returned error can't find the container with id a2bc7afdc85a97428028c2548b534d7d6d2c9f043dc74309f001e58480e16e56 Oct 30 06:00:31.058805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:31.058770 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5"] Oct 30 06:00:31.061709 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:31.061678 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:31.061790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:31.061746 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:31.061790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:31.061777 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:00:31.498289 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:31.498249 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" event={"ID":"715162c8-0a95-4c0d-b15e-3e57c3b707f3","Type":"ContainerStarted","Data":"3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4"} Oct 30 06:00:31.498544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:31.498315 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" event={"ID":"715162c8-0a95-4c0d-b15e-3e57c3b707f3","Type":"ContainerStarted","Data":"a2bc7afdc85a97428028c2548b534d7d6d2c9f043dc74309f001e58480e16e56"} Oct 30 06:00:31.556252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:31.556182 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" podStartSLOduration=1.5561651890000001 podStartE2EDuration="1.556165189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:31.554827212 +0000 UTC m=+1791.158516470" watchObservedRunningTime="2026-10-30 06:00:31.556165189 +0000 UTC m=+1791.159854446" Oct 30 06:00:31.738579 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:31.738542 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:32.742734 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:32.742692 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:00:32.743258 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:32.742766 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:00:33.470521 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:33.470489 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:00:33.750754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:33.750661 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:00:33.750754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:33.750719 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:00:34.740957 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:34.740915 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:00:34.741174 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:34.740989 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:00:35.506068 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:35.500648 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:00:35.738483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:35.736600 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:00:35.738483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:35.736693 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:00:36.734483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:36.733942 5072 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 06:00:36.734483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:36.734029 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 06:00:37.522524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.518394 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4"] Oct 30 06:00:37.527484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.524946 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.556730 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.553755 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4"] Oct 30 06:00:37.556730 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.554923 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1dbaf6-serving-cert\"" Oct 30 06:00:37.597857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.597750 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5crbp\" (UniqueName: \"kubernetes.io/projected/d706291d-60bd-427e-81b2-98523f2f5d2f-kube-api-access-5crbp\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.597857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.597819 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-metrics-certs\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.598156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.598005 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.598223 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.598200 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d706291d-60bd-427e-81b2-98523f2f5d2f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.700277 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.700225 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5crbp\" (UniqueName: \"kubernetes.io/projected/d706291d-60bd-427e-81b2-98523f2f5d2f-kube-api-access-5crbp\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.700546 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.700312 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-metrics-certs\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.700546 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.700536 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.700696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.700666 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d706291d-60bd-427e-81b2-98523f2f5d2f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.701398 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.701373 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d706291d-60bd-427e-81b2-98523f2f5d2f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.703349 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.703318 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-metrics-certs\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.703834 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.703809 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.723890 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.723844 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5crbp\" (UniqueName: \"kubernetes.io/projected/d706291d-60bd-427e-81b2-98523f2f5d2f-kube-api-access-5crbp\") pod \"maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.874044 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.873981 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:37.886906 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.886864 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:37.889427 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.889401 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:37.911184 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:37.902956 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:00:38.371426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:38.371396 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4"] Oct 30 06:00:38.372514 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:00:38.372489 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd706291d_60bd_427e_81b2_98523f2f5d2f.slice/crio-b055b0afdb5f3364d44e7985c1650df7c66fc783a455ab5d785cd6172c1a1c01 WatchSource:0}: Error finding container b055b0afdb5f3364d44e7985c1650df7c66fc783a455ab5d785cd6172c1a1c01: Status 404 returned error can't find the container with id b055b0afdb5f3364d44e7985c1650df7c66fc783a455ab5d785cd6172c1a1c01 Oct 30 06:00:38.770829 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:38.770762 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" event={"ID":"d706291d-60bd-427e-81b2-98523f2f5d2f","Type":"ContainerStarted","Data":"b055b0afdb5f3364d44e7985c1650df7c66fc783a455ab5d785cd6172c1a1c01"} Oct 30 06:00:39.780676 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:39.780645 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" event={"ID":"d706291d-60bd-427e-81b2-98523f2f5d2f","Type":"ContainerStarted","Data":"d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70"} Oct 30 06:00:39.782242 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:39.781732 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:39.824692 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:39.824636 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" podStartSLOduration=2.475526167 podStartE2EDuration="2.8246221s" totalImagesPullingTime="349.095933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:39.820616246 +0000 UTC m=+1799.424305503" watchObservedRunningTime="2026-10-30 06:00:39.8246221 +0000 UTC m=+1799.428311357" Oct 30 06:00:41.121710 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.121667 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl"] Oct 30 06:00:41.127894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.127862 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.152117 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.152088 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f28e2d-serving-cert\"" Oct 30 06:00:41.155877 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.155840 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl"] Oct 30 06:00:41.237027 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.236983 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.237220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.237197 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-metrics-certs\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.237302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.237275 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7c0d1e-a4c9-4943-9454-c09687532703-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.237386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.237369 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67t7\" (UniqueName: \"kubernetes.io/projected/fe7c0d1e-a4c9-4943-9454-c09687532703-kube-api-access-c67t7\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.341729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.341685 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-metrics-certs\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.341949 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.341808 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7c0d1e-a4c9-4943-9454-c09687532703-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.341949 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.341925 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67t7\" (UniqueName: \"kubernetes.io/projected/fe7c0d1e-a4c9-4943-9454-c09687532703-kube-api-access-c67t7\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.342163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.342081 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.363355 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.352931 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7c0d1e-a4c9-4943-9454-c09687532703-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.363355 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:00:41.360054 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-serving-cert: secret "maas-api-e2e-inf-a-f28e2d-serving-cert" not found Oct 30 06:00:41.363355 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:00:41.360142 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls podName:fe7c0d1e-a4c9-4943-9454-c09687532703 nodeName:}" failed. No retries permitted until 2026-10-30 06:00:41.860120193 +0000 UTC m=+1801.463809432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls") pod "maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" (UID: "fe7c0d1e-a4c9-4943-9454-c09687532703") : secret "maas-api-e2e-inf-a-f28e2d-serving-cert" not found Oct 30 06:00:41.368265 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.368234 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-metrics-certs\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.388242 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.388150 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67t7\" (UniqueName: \"kubernetes.io/projected/fe7c0d1e-a4c9-4943-9454-c09687532703-kube-api-access-c67t7\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.869966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.869935 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:41.872704 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:41.872678 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls\") pod \"maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:42.046817 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:42.046783 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:42.364452 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:42.364422 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl"] Oct 30 06:00:42.365202 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:00:42.365169 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7c0d1e_a4c9_4943_9454_c09687532703.slice/crio-74b300f58a9bc1d5b6f0d2670cfa88639659b61961f13ab0d4d1d9c5d79ed2c6 WatchSource:0}: Error finding container 74b300f58a9bc1d5b6f0d2670cfa88639659b61961f13ab0d4d1d9c5d79ed2c6: Status 404 returned error can't find the container with id 74b300f58a9bc1d5b6f0d2670cfa88639659b61961f13ab0d4d1d9c5d79ed2c6 Oct 30 06:00:42.806902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:42.806867 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" event={"ID":"fe7c0d1e-a4c9-4943-9454-c09687532703","Type":"ContainerStarted","Data":"74b300f58a9bc1d5b6f0d2670cfa88639659b61961f13ab0d4d1d9c5d79ed2c6"} Oct 30 06:00:43.109424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.109391 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr"] Oct 30 06:00:43.114590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.114566 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.157066 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.142531 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-765de9-serving-cert\"" Oct 30 06:00:43.184104 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.184070 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr"] Oct 30 06:00:43.203201 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.203163 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-maas-api-tls\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.203368 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.203322 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkr6v\" (UniqueName: \"kubernetes.io/projected/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-kube-api-access-zkr6v\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.203513 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.203381 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-metrics-certs\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.203606 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.203585 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.305409 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.305368 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.305617 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.305601 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-maas-api-tls\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.305724 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.305709 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkr6v\" (UniqueName: \"kubernetes.io/projected/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-kube-api-access-zkr6v\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.305796 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.305776 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-metrics-certs\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.306127 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.306102 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.308289 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.308270 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-metrics-certs\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.308372 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.308332 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-maas-api-tls\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.328359 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.328321 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkr6v\" (UniqueName: \"kubernetes.io/projected/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-kube-api-access-zkr6v\") pod \"maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.474293 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.470677 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:43.820257 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.820165 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" event={"ID":"fe7c0d1e-a4c9-4943-9454-c09687532703","Type":"ContainerStarted","Data":"77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f"} Oct 30 06:00:43.820405 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.820276 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:43.851637 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:00:43.851606 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37dc5ac1_6e11_4bd0_b1d2_4fd52eb0baf5.slice/crio-a673244d6257c642062b6b0c91d1983a0fc648f00c909b17d4d99dc88d8a78ca WatchSource:0}: Error finding container a673244d6257c642062b6b0c91d1983a0fc648f00c909b17d4d99dc88d8a78ca: Status 404 returned error can't find the container with id a673244d6257c642062b6b0c91d1983a0fc648f00c909b17d4d99dc88d8a78ca Oct 30 06:00:43.852615 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:43.852589 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr"] Oct 30 06:00:44.831153 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:44.831085 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" event={"ID":"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5","Type":"ContainerStarted","Data":"3d004fe4ad097ec9e597627771bbac46e39b8fb5b9281af1e4f04945bb14dd74"} Oct 30 06:00:44.831153 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:44.831148 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" event={"ID":"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5","Type":"ContainerStarted","Data":"a673244d6257c642062b6b0c91d1983a0fc648f00c909b17d4d99dc88d8a78ca"} Oct 30 06:00:44.831807 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:44.831213 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:00:44.871548 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:44.871495 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" podStartSLOduration=1.536372394 podStartE2EDuration="1.871481667s" totalImagesPullingTime="335.109273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:44.869996076 +0000 UTC m=+1804.473685334" watchObservedRunningTime="2026-10-30 06:00:44.871481667 +0000 UTC m=+1804.475170921" Oct 30 06:00:44.872154 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:44.872127 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" podStartSLOduration=3.444855682 podStartE2EDuration="3.872121842s" totalImagesPullingTime="427.26616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:43.859711083 +0000 UTC m=+1803.463400341" watchObservedRunningTime="2026-10-30 06:00:44.872121842 +0000 UTC m=+1804.475811134" Oct 30 06:00:45.793392 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:45.793358 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:00:47.755832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:47.755804 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:00:47.805961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:47.805932 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:00:47.819476 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:47.819438 5072 kubelet.go:1689] "Image garbage collection succeeded" Oct 30 06:00:49.837883 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:49.837808 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:00:50.848452 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:00:50.848421 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:01:00.711747 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:01:00.711704 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:01:06.616486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:06.615594 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:06.666488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:06.665724 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 06:01:07.805868 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:07.805574 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:08.167485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:08.166953 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:08.616544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:08.613739 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 06:01:08.683488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:08.659890 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 06:01:08.839498 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:08.838592 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:09.116493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.112591 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:09.225496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.224587 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:09.225496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.225051 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 06:01:09.703737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.637621 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:09.703737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.638200 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 06:01:09.703737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.678574 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:09.703737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.679213 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:01:09.844530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.834926 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 06:01:10.025580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.882554 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:10.025580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.883431 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 06:01:10.025580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:09.960107 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:10.233192 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:10.233139 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:10.698591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:10.693196 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:10.997524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:10.993667 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 06:01:11.029900 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:11.029855 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:11.233786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:11.207145 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:11.255799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:11.255711 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 06:01:11.258635 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:11.258611 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:01:11.271487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:11.271055 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 06:01:11.467797 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:11.467764 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:01:12.149485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:12.102346 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 06:01:12.151044 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:12.115913 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 06:01:12.151774 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:12.115966 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": context deadline exceeded" Oct 30 06:01:12.197808 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:12.116235 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 06:01:12.223535 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:12.116266 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/health\": context deadline exceeded" Oct 30 06:01:12.269157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:12.269095 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:13.107590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:13.107557 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 06:01:13.163480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:13.151567 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:13.163480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:13.151953 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 06:01:13.163480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:13.153328 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 06:01:13.208982 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:13.207112 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 06:01:13.208982 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:13.207992 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:01:13.285252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:13.284301 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:01:13.301484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:13.297060 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 06:01:13.301484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:13.297810 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:01:16.413948 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:16.413596 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:01:16.413948 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:16.413675 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:17.422661 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:17.422610 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:01:17.423286 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:17.422718 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:22.774025 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.773935 5072 generic.go:356] "Generic (PLEG): container finished" podID="9375e831-d43b-41c9-af62-082c05de351e" containerID="3018b74b597349dba960d173f998672bf7f0ca3c1db6a79114fb5fa3c81d7046" exitCode=0 Oct 30 06:01:22.774025 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.774004 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" event={"ID":"9375e831-d43b-41c9-af62-082c05de351e","Type":"ContainerDied","Data":"3018b74b597349dba960d173f998672bf7f0ca3c1db6a79114fb5fa3c81d7046"} Oct 30 06:01:22.774656 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.774565 5072 scope.go:122] "RemoveContainer" containerID="3018b74b597349dba960d173f998672bf7f0ca3c1db6a79114fb5fa3c81d7046" Oct 30 06:01:22.775834 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.775813 5072 generic.go:356] "Generic (PLEG): container finished" podID="45cfb151-b682-400e-a174-f37d2c65f08a" containerID="f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad" exitCode=0 Oct 30 06:01:22.775939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.775918 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" event={"ID":"45cfb151-b682-400e-a174-f37d2c65f08a","Type":"ContainerDied","Data":"f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad"} Oct 30 06:01:22.776286 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.776269 5072 scope.go:122] "RemoveContainer" containerID="f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad" Oct 30 06:01:22.776687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.776650 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.776793 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.776740 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.776841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.776793 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.777751 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.777732 5072 generic.go:356] "Generic (PLEG): container finished" podID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerID="5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234" exitCode=0 Oct 30 06:01:22.777864 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.777805 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" event={"ID":"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c","Type":"ContainerDied","Data":"5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234"} Oct 30 06:01:22.778070 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.778045 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.778191 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.778109 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.778191 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.778154 5072 scope.go:122] "RemoveContainer" containerID="5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234" Oct 30 06:01:22.778312 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.778158 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.779989 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.779957 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.780095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.780033 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.780095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.780084 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.780407 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.780384 5072 generic.go:356] "Generic (PLEG): container finished" podID="0e614f11-3285-4826-aec4-ffa24b359348" containerID="15e9fb0be3f0f9847ff14b84421814090d645fcacdbca5206c521b23db44e68f" exitCode=0 Oct 30 06:01:22.780569 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.780416 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" event={"ID":"0e614f11-3285-4826-aec4-ffa24b359348","Type":"ContainerDied","Data":"15e9fb0be3f0f9847ff14b84421814090d645fcacdbca5206c521b23db44e68f"} Oct 30 06:01:22.780874 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.780820 5072 scope.go:122] "RemoveContainer" containerID="15e9fb0be3f0f9847ff14b84421814090d645fcacdbca5206c521b23db44e68f" Oct 30 06:01:22.783053 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.783030 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.783150 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.783109 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:22.783213 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:22.783145 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:23.801165 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:23.801135 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" event={"ID":"9375e831-d43b-41c9-af62-082c05de351e","Type":"ContainerStarted","Data":"e98e6f2a57fa95e5b6fc673b41cf1df3bb39622d178f8a28da5cd0fc7071849e"} Oct 30 06:01:23.804576 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:23.804071 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" event={"ID":"45cfb151-b682-400e-a174-f37d2c65f08a","Type":"ContainerStarted","Data":"4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad"} Oct 30 06:01:23.807798 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:23.807426 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" event={"ID":"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c","Type":"ContainerStarted","Data":"8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b"} Oct 30 06:01:23.811599 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:23.810208 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" event={"ID":"0e614f11-3285-4826-aec4-ffa24b359348","Type":"ContainerStarted","Data":"0c62c319a143899c3d2757e8b8d16071ef59d7240b40533e9c7f88cb9ca8f489"} Oct 30 06:01:23.961428 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:23.961385 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 06:01:23.962342 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:23.962321 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:23.962417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:23.962366 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:24.317696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.312631 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:24.317696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.312692 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:24.317696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.313079 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 06:01:24.357056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.357018 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 06:01:24.357927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.357902 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:24.358128 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.358105 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:24.798664 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.798602 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 06:01:24.815170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.798974 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:24.834655 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.834606 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:24.966944 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.966909 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:24.967230 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:24.967201 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:25.312965 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.312918 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:25.313332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.313310 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:25.313790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.313233 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:25.393482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.392936 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:25.393482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.393032 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:25.483482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.476557 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:25.810700 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.810662 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:25.810969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.810937 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:26.044484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.992556 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:26.044484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:25.992617 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:26.181152 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.181047 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": context deadline exceeded" Oct 30 06:01:26.205929 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.205884 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 06:01:26.276707 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.274843 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 06:01:26.325682 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.304672 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:26.386485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.341387 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:26.386485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.370339 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 06:01:26.386485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.347594 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:26.386485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.371444 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:26.415410 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.415375 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:26.415715 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.415685 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:26.487921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.487823 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 06:01:26.488961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.488937 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:01:26.828519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.828411 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:26.828786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.828755 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:27.085536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:26.963544 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:27.085536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.085351 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:27.215612 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.215563 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:27.290953 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.290903 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 06:01:27.305920 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.305879 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:27.337536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.311302 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:27.351492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.343572 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:27.351492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.313854 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 06:01:27.351492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.344167 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:01:27.351492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.344246 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 06:01:27.351492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.344309 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:01:27.374660 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.374605 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 06:01:27.416777 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.416742 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:27.417055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.417020 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:27.508925 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.508885 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:27.858475 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.823682 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:27.858475 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.823759 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:27.861255 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:27.861229 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 06:01:28.007582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.007544 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:28.025626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.025587 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:28.026259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.013483 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:28.060728 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.034101 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:01:28.178578 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.178533 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:28.216780 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.216736 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 06:01:28.306490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.306380 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 06:01:28.348544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.330311 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:28.401313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.400605 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 06:01:28.442844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.436723 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:28.442844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.436781 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:28.445071 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.443728 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:28.445071 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.443784 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:28.828492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.827272 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:28.828492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.827346 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:28.965483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.939715 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:01:28.965483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.964079 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:01:28.968724 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:28.968690 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:29.202525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:29.199575 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:29.428520 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:29.428481 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:29.431701 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:29.431641 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:29.537946 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:29.537842 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 06:01:29.880651 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:29.724987 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:29.889636 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:29.889583 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:30.155058 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.154954 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 06:01:30.156273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.156223 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:30.156534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.156491 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:30.160024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.160000 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:30.160261 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.160220 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:30.327920 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.327867 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:30.362488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.350145 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:30.368583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.368531 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:30.402488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.399180 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 06:01:30.426702 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.426606 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:30.426998 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.426948 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:30.498640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.498586 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:30.796414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.788057 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:30.796768 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.796730 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:30.797319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.797289 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:30.859595 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.859551 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:30.999130 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.999091 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:30.999384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:30.999360 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:31.176139 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:01:31.174659 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:01:31.230176 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.227570 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:31.230176 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.228614 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:31.273485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.266671 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:31.350408 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.350372 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:31.357485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.351538 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:31.358266 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.358237 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:31.358542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.358510 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:31.397590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.397551 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 06:01:31.515633 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.515531 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:31.516410 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.516386 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:01:31.744627 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.744601 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:01:31.785078 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.784979 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:31.785361 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.785312 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:31.810753 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.810707 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 06:01:31.826161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.826123 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:01:31.937742 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.861428 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:31.965246 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.965220 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:01:31.983880 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.967374 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:31.984152 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.984121 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:31.984691 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:31.984664 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:32.232997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.232684 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:32.322485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.319300 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:32.349522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.340277 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 06:01:32.349522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.339758 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:32.349522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.340420 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:32.422519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.388719 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 06:01:32.430741 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.408582 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:32.430741 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.430663 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:32.817716 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.817589 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:32.818110 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.817982 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:32.939484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.938561 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:32.965356 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.965308 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 06:01:32.966002 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.965971 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:32.993312 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:32.993268 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:33.330484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.312577 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:33.330484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.312653 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:33.355484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.342759 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:33.381485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.369676 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:33.381485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.372249 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:01:33.381485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.374034 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:33.381485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.374079 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:33.456483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.451547 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 06:01:33.509488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.506842 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 30 06:01:33.751911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.749996 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 06:01:33.790744 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.790707 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:33.791006 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.790980 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:33.939639 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.939583 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 06:01:33.961859 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.961822 5072 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:01:33.962124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.962099 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 06:01:34.000486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:33.974427 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:34.020994 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.020910 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:01:34.314966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.314874 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:01:34.354776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.354734 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 06:01:34.357495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.346011 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:34.393342 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.365744 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:34.393342 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.366110 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:34.393342 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.392132 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 06:01:34.471952 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.471908 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 06:01:34.556477 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.553071 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:01:34.871482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.863057 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 06:01:34.999696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.946371 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:34.999696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:34.946435 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:35.109897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.109841 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 06:01:35.181543 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.177820 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.404516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.403446 5072 patch_prober.go:29] interesting pod/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 06:01:35.404516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.403544 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 06:01:35.413801 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.406210 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": context deadline exceeded" Oct 30 06:01:35.474157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.474060 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.602403 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:01:35.602363 5072 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.043s" Oct 30 06:01:35.602793 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.602775 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:01:35.603515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.603495 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 06:01:35.609630 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.609598 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.684490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.640907 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 06:01:35.699061 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.699031 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 06:01:35.922731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:35.922685 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:36.042482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.025901 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 06:01:36.162698 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.162663 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 06:01:36.163810 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.163774 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 06:01:36.163948 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.163834 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 06:01:36.191431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.178056 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 06:01:36.191431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.187577 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Oct 30 06:01:36.271079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.271052 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:01:36.360523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.360494 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:01:36.375349 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.375318 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 06:01:36.377789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.377768 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 06:01:36.385760 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.384990 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 06:01:36.824496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.823275 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 06:01:36.835504 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:36.828677 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 06:01:40.860415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:40.860034 5072 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:01:40.860415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:40.860102 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:43.365270 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.365235 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc"] Oct 30 06:01:43.376291 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.376201 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.422788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.422746 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7dfcb7-openshift-default-dockercfg-h8vd6\"" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.438184 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-certs\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.438367 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.438404 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-credential-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.438555 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mmf\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-kube-api-access-r5mmf\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.438600 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e2d554-5320-439f-a7cb-133f6c6e4780-istiod-ca-cert\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.438738 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-podinfo\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.438815 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-data\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.438843 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-token\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.438928 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-envoy\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.463559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.460661 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc"] Oct 30 06:01:43.625534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.625084 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-envoy\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.625534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.625244 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-certs\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.625534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.625496 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.625815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.625552 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-credential-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.625815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.625739 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mmf\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-kube-api-access-r5mmf\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.625815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.625800 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e2d554-5320-439f-a7cb-133f6c6e4780-istiod-ca-cert\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.625991 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.625949 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-podinfo\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.632489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.628085 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-credential-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.632489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.630478 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e2d554-5320-439f-a7cb-133f6c6e4780-istiod-ca-cert\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.632489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.630931 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-certs\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.632489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.631530 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.655479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.642711 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-data\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.655479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.642771 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-token\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.655479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.643691 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-data\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.674665 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.659696 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-envoy\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.681483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.677895 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-podinfo\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.708962 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.708927 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-token\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.719483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.713439 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mmf\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-kube-api-access-r5mmf\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:43.766791 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:43.766752 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:44.323097 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:44.323064 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc"] Oct 30 06:01:44.324183 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:01:44.324143 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e2d554_5320_439f_a7cb_133f6c6e4780.slice/crio-6b3903863483d43e0bbba7924bf2cedd7a7de39f234a366f0a4a956e6f56ff54 WatchSource:0}: Error finding container 6b3903863483d43e0bbba7924bf2cedd7a7de39f234a366f0a4a956e6f56ff54: Status 404 returned error can't find the container with id 6b3903863483d43e0bbba7924bf2cedd7a7de39f234a366f0a4a956e6f56ff54 Oct 30 06:01:44.328232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:44.328194 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:44.328349 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:44.328294 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:44.328349 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:44.328344 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:44.824453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:44.824400 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" event={"ID":"e4e2d554-5320-439f-a7cb-133f6c6e4780","Type":"ContainerStarted","Data":"32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96"} Oct 30 06:01:44.826797 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:44.824988 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" event={"ID":"e4e2d554-5320-439f-a7cb-133f6c6e4780","Type":"ContainerStarted","Data":"6b3903863483d43e0bbba7924bf2cedd7a7de39f234a366f0a4a956e6f56ff54"} Oct 30 06:01:44.890880 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:44.890816 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podStartSLOduration=1.890798014 podStartE2EDuration="1.890798014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:44.87715815 +0000 UTC m=+1864.480847408" watchObservedRunningTime="2026-10-30 06:01:44.890798014 +0000 UTC m=+1864.494487272" Oct 30 06:01:45.232510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.231580 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb"] Oct 30 06:01:45.769171 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.769143 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:45.776393 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.776360 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:45.837979 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.837939 5072 generic.go:356] "Generic (PLEG): container finished" podID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerID="c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03" exitCode=0 Oct 30 06:01:45.838520 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.837987 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" event={"ID":"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a","Type":"ContainerDied","Data":"c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03"} Oct 30 06:01:45.838520 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.838292 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:45.838936 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.838916 5072 scope.go:122] "RemoveContainer" containerID="c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03" Oct 30 06:01:45.840449 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.840205 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:01:45.841427 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.841389 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:45.841580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.841498 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:45.841580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:45.841546 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:01:46.853015 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:46.852963 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" event={"ID":"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a","Type":"ContainerStarted","Data":"c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8"} Oct 30 06:01:47.190923 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:47.190847 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 06:01:47.191095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:47.191065 5072 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:01:47.191151 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:47.191123 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:01:48.191124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:48.191092 5072 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:01:48.191637 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:48.191142 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:01:49.191202 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:49.191170 5072 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:01:49.191643 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:49.191239 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:01:50.191595 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:50.191562 5072 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:01:50.192051 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:50.191631 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:01:51.191784 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:51.191750 5072 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:01:51.192202 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:51.191824 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:01:52.192012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:52.191982 5072 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:01:52.192440 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:52.192032 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:01:53.191080 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:53.191038 5072 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 06:01:53.191301 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:53.191119 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 06:01:54.197927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:54.197893 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 06:01:54.198559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:54.198443 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 06:01:54.199491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:01:54.199443 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 06:02:00.269999 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.269930 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" containerID="cri-o://ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9" gracePeriod=45 Oct 30 06:02:00.343831 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.343792 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.142:8000/ready\": dial tcp 10.132.0.142:8000: connect: connection refused" Oct 30 06:02:00.601430 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.601402 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 06:02:00.717122 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:02:00.717076 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:02:00.784045 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.784007 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-model-cache\") pod \"5263d63d-6812-4507-aead-a17dd19cef3b\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " Oct 30 06:02:00.784245 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.784145 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-dshm\") pod \"5263d63d-6812-4507-aead-a17dd19cef3b\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " Oct 30 06:02:00.784321 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.784258 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-model-cache" pod "5263d63d-6812-4507-aead-a17dd19cef3b" (UID: "5263d63d-6812-4507-aead-a17dd19cef3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:00.784321 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.784310 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-tmp-dir\") pod \"5263d63d-6812-4507-aead-a17dd19cef3b\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " Oct 30 06:02:00.784522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.784449 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-home\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-home\") pod \"5263d63d-6812-4507-aead-a17dd19cef3b\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " Oct 30 06:02:00.784617 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.784540 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-tmp-dir" pod "5263d63d-6812-4507-aead-a17dd19cef3b" (UID: "5263d63d-6812-4507-aead-a17dd19cef3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:00.784617 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.784545 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5263d63d-6812-4507-aead-a17dd19cef3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5263d63d-6812-4507-aead-a17dd19cef3b-tls-certs\") pod \"5263d63d-6812-4507-aead-a17dd19cef3b\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " Oct 30 06:02:00.784617 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.784607 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5263d63d-6812-4507-aead-a17dd19cef3b-kube-api-access-qwgrb\" (UniqueName: \"kubernetes.io/projected/5263d63d-6812-4507-aead-a17dd19cef3b-kube-api-access-qwgrb\") pod \"5263d63d-6812-4507-aead-a17dd19cef3b\" (UID: \"5263d63d-6812-4507-aead-a17dd19cef3b\") " Oct 30 06:02:00.784790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.784701 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-home" pod "5263d63d-6812-4507-aead-a17dd19cef3b" (UID: "5263d63d-6812-4507-aead-a17dd19cef3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:00.785302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.785281 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:00.785302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.785303 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:00.785658 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.785330 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:00.786821 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.786795 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5263d63d-6812-4507-aead-a17dd19cef3b-tls-certs" pod "5263d63d-6812-4507-aead-a17dd19cef3b" (UID: "5263d63d-6812-4507-aead-a17dd19cef3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:00.786913 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.786799 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-dshm" pod "5263d63d-6812-4507-aead-a17dd19cef3b" (UID: "5263d63d-6812-4507-aead-a17dd19cef3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:00.786962 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.786930 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5263d63d-6812-4507-aead-a17dd19cef3b-kube-api-access-qwgrb" pod "5263d63d-6812-4507-aead-a17dd19cef3b" (UID: "5263d63d-6812-4507-aead-a17dd19cef3b"). InnerVolumeSpecName "kube-api-access-qwgrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:00.890487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.888060 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5263d63d-6812-4507-aead-a17dd19cef3b-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:00.890487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.888095 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwgrb\" (UniqueName: \"kubernetes.io/projected/5263d63d-6812-4507-aead-a17dd19cef3b-kube-api-access-qwgrb\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:00.890487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:00.888110 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5263d63d-6812-4507-aead-a17dd19cef3b-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:01.179107 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:01.179008 5072 generic.go:356] "Generic (PLEG): container finished" podID="5263d63d-6812-4507-aead-a17dd19cef3b" containerID="ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9" exitCode=0 Oct 30 06:02:01.179314 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:01.179288 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" event={"ID":"5263d63d-6812-4507-aead-a17dd19cef3b","Type":"ContainerDied","Data":"ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9"} Oct 30 06:02:01.179393 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:01.179336 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" event={"ID":"5263d63d-6812-4507-aead-a17dd19cef3b","Type":"ContainerDied","Data":"c6aba4e61e890f5b93426fcef1ad36789b3e651d378b884b5ced016715049e59"} Oct 30 06:02:01.179393 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:01.179364 5072 scope.go:122] "RemoveContainer" containerID="ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9" Oct 30 06:02:01.179592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:01.179576 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb" Oct 30 06:02:01.238708 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:01.238676 5072 scope.go:122] "RemoveContainer" containerID="ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9" Oct 30 06:02:01.239413 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:02:01.239381 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9\": container with ID starting with ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9 not found: ID does not exist" containerID="ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9" Oct 30 06:02:01.239627 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:01.239598 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9"} err="failed to get container status \"ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9\": rpc error: code = NotFound desc = could not find container \"ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9\": container with ID starting with ce472deac905ecd20214ff7b585535be636d99218eac31cfd5a74c5647162bc9 not found: ID does not exist" Oct 30 06:02:01.296667 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:01.296633 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb"] Oct 30 06:02:01.306986 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:01.306959 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-39147c/test-model-39147c-kserve-7ccd6c46df-wxbxb"] Oct 30 06:02:02.551369 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:02.551335 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" path="/var/lib/kubelet/pods/5263d63d-6812-4507-aead-a17dd19cef3b/volumes" Oct 30 06:02:13.104629 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:13.104581 5072 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:02:13.105164 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:13.104638 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:14.369696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:14.369665 5072 generic.go:356] "Generic (PLEG): container finished" podID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerID="3f785a16d41b9b6ce314f1ae38ee73256bff6f66e43350b7874df874d55f09f5" exitCode=0 Oct 30 06:02:14.370179 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:14.369740 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" event={"ID":"52f96c85-f7ff-4335-85e9-2a998e9b8b66","Type":"ContainerDied","Data":"3f785a16d41b9b6ce314f1ae38ee73256bff6f66e43350b7874df874d55f09f5"} Oct 30 06:02:14.370277 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:14.370208 5072 scope.go:122] "RemoveContainer" containerID="3f785a16d41b9b6ce314f1ae38ee73256bff6f66e43350b7874df874d55f09f5" Oct 30 06:02:14.372209 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:14.372183 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:02:14.372280 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:14.372249 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:02:14.372319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:14.372281 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:02:15.389059 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:15.389005 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" event={"ID":"52f96c85-f7ff-4335-85e9-2a998e9b8b66","Type":"ContainerStarted","Data":"fc6c2a4d4657277876509ccad3946bcebcacc21dbf63b7b1b2660e2b711d983d"} Oct 30 06:02:15.567964 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:15.567928 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb"] Oct 30 06:02:15.568179 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:15.568145 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" podUID="af170278-15d4-44d0-9f2c-3a91ad682991" containerName="maas-api" containerID="cri-o://30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88" gracePeriod=30 Oct 30 06:02:16.236197 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.234398 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:02:16.248800 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.248762 5072 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 06:02:16.251002 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.250967 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 06:02:16.439501 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.423064 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 06:02:16.553577 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.551548 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-metrics-certs\") pod \"af170278-15d4-44d0-9f2c-3a91ad682991\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " Oct 30 06:02:16.553577 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.551626 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af170278-15d4-44d0-9f2c-3a91ad682991-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af170278-15d4-44d0-9f2c-3a91ad682991-openshift-service-ca\") pod \"af170278-15d4-44d0-9f2c-3a91ad682991\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " Oct 30 06:02:16.553577 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.551736 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af170278-15d4-44d0-9f2c-3a91ad682991-kube-api-access-sszlz\" (UniqueName: \"kubernetes.io/projected/af170278-15d4-44d0-9f2c-3a91ad682991-kube-api-access-sszlz\") pod \"af170278-15d4-44d0-9f2c-3a91ad682991\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " Oct 30 06:02:16.553577 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.552074 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-maas-api-tls\") pod \"af170278-15d4-44d0-9f2c-3a91ad682991\" (UID: \"af170278-15d4-44d0-9f2c-3a91ad682991\") " Oct 30 06:02:16.560732 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.559720 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af170278-15d4-44d0-9f2c-3a91ad682991-openshift-service-ca" pod "af170278-15d4-44d0-9f2c-3a91ad682991" (UID: "af170278-15d4-44d0-9f2c-3a91ad682991"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:16.581272 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.581223 5072 generic.go:356] "Generic (PLEG): container finished" podID="af170278-15d4-44d0-9f2c-3a91ad682991" containerID="30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88" exitCode=0 Oct 30 06:02:16.583343 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.583310 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" Oct 30 06:02:16.672838 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.618869 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af170278-15d4-44d0-9f2c-3a91ad682991-kube-api-access-sszlz" pod "af170278-15d4-44d0-9f2c-3a91ad682991" (UID: "af170278-15d4-44d0-9f2c-3a91ad682991"). InnerVolumeSpecName "kube-api-access-sszlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:16.682121 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.682090 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af170278-15d4-44d0-9f2c-3a91ad682991-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:16.682491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.682473 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sszlz\" (UniqueName: \"kubernetes.io/projected/af170278-15d4-44d0-9f2c-3a91ad682991-kube-api-access-sszlz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:16.746753 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.746720 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-metrics-certs" pod "af170278-15d4-44d0-9f2c-3a91ad682991" (UID: "af170278-15d4-44d0-9f2c-3a91ad682991"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:16.747044 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.747026 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-maas-api-tls" pod "af170278-15d4-44d0-9f2c-3a91ad682991" (UID: "af170278-15d4-44d0-9f2c-3a91ad682991"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:16.765206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.764842 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" event={"ID":"af170278-15d4-44d0-9f2c-3a91ad682991","Type":"ContainerDied","Data":"30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88"} Oct 30 06:02:16.765206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.764919 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb" event={"ID":"af170278-15d4-44d0-9f2c-3a91ad682991","Type":"ContainerDied","Data":"96f9db896d69b8142c1049982004e195a0e5186fc2538994c22b18bbdabf07a9"} Oct 30 06:02:16.765206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.764949 5072 scope.go:122] "RemoveContainer" containerID="30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88" Oct 30 06:02:16.799614 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.799577 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:16.806534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.799836 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af170278-15d4-44d0-9f2c-3a91ad682991-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:16.893292 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.893267 5072 scope.go:122] "RemoveContainer" containerID="30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88" Oct 30 06:02:16.893923 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:02:16.893897 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88\": container with ID starting with 30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88 not found: ID does not exist" containerID="30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88" Oct 30 06:02:16.894098 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.894073 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88"} err="failed to get container status \"30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88\": rpc error: code = NotFound desc = could not find container \"30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88\": container with ID starting with 30502e94908e974f46d7be0b71cc101b1b337ce18a648376999ed269f4a9cd88 not found: ID does not exist" Oct 30 06:02:16.957076 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.957035 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb"] Oct 30 06:02:16.961501 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:16.961471 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b3e625-597b869bdd-lrhhb"] Oct 30 06:02:17.203499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:17.203453 5072 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 06:02:17.203655 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:17.203530 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 06:02:18.201938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:18.201908 5072 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 06:02:18.202323 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:18.201956 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 06:02:18.545228 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:18.544495 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af170278-15d4-44d0-9f2c-3a91ad682991" path="/var/lib/kubelet/pods/af170278-15d4-44d0-9f2c-3a91ad682991/volumes" Oct 30 06:02:19.202267 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.202234 5072 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 06:02:19.203971 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.203938 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 06:02:19.280584 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.280543 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds"] Oct 30 06:02:19.282712 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.282657 5072 cpu_manager.go:414] "containerMap: removing container" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" Oct 30 06:02:19.282712 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.282688 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" Oct 30 06:02:19.282921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.282828 5072 cpu_manager.go:414] "containerMap: removing container" podUID="af170278-15d4-44d0-9f2c-3a91ad682991" containerName="maas-api" Oct 30 06:02:19.282921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.282837 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af170278-15d4-44d0-9f2c-3a91ad682991" containerName="maas-api" Oct 30 06:02:19.283230 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.283203 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="af170278-15d4-44d0-9f2c-3a91ad682991" containerName="maas-api" Oct 30 06:02:19.283317 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.283237 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5263d63d-6812-4507-aead-a17dd19cef3b" containerName="main" Oct 30 06:02:19.296311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.295517 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds"] Oct 30 06:02:19.296730 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.296712 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.343483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.338771 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-02ceed-serving-cert\"" Oct 30 06:02:19.454534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.406203 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.457186 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.457149 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6pcc\" (UniqueName: \"kubernetes.io/projected/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-kube-api-access-c6pcc\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.458190 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.458163 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.458376 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.458357 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.565871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.565830 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.566617 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.566590 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6pcc\" (UniqueName: \"kubernetes.io/projected/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-kube-api-access-c6pcc\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.566870 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.566850 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.567092 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.567075 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.568471 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.568429 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.590971 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.590937 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.593052 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.593016 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.594242 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.593958 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6pcc\" (UniqueName: \"kubernetes.io/projected/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-kube-api-access-c6pcc\") pod \"maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:19.800594 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:19.800498 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:20.161097 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:20.161064 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds"] Oct 30 06:02:20.164968 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:20.164945 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:02:20.202046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:20.202011 5072 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 06:02:20.202208 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:20.202070 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 06:02:20.649448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:20.649404 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" event={"ID":"a8b1bd51-0d34-4d80-b0f7-80024906a8a5","Type":"ContainerStarted","Data":"c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252"} Oct 30 06:02:20.650648 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:20.649527 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" event={"ID":"a8b1bd51-0d34-4d80-b0f7-80024906a8a5","Type":"ContainerStarted","Data":"68a0a27718c651b5a545632a9d63726f95f4ff9b4bd1f420e31dd1df421d535d"} Oct 30 06:02:20.650648 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:20.649681 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:20.724573 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:20.711852 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" podStartSLOduration=1.402896562 podStartE2EDuration="1.711828133s" totalImagesPullingTime="308.931571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:20.690148138 +0000 UTC m=+1900.293837493" watchObservedRunningTime="2026-10-30 06:02:20.711828133 +0000 UTC m=+1900.315517395" Oct 30 06:02:21.206675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:21.204677 5072 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 06:02:21.206675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:21.204735 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 06:02:22.208208 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:22.208177 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:02:22.208739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:22.208721 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:02:22.209586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:22.209565 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:02:24.022913 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:24.022880 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l"] Oct 30 06:02:24.023573 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:24.023519 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" containerID="cri-o://6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08" gracePeriod=30 Oct 30 06:02:26.687013 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:26.686982 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:02:29.338416 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.338388 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 06:02:29.392380 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392338 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-certs\") pod \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " Oct 30 06:02:29.392380 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392377 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istiod-ca-cert\") pod \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " Oct 30 06:02:29.392659 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392447 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-data\") pod \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " Oct 30 06:02:29.392659 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392523 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-credential-socket\") pod \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " Oct 30 06:02:29.392659 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392571 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-kube-api-access-44vvx\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-kube-api-access-44vvx\") pod \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " Oct 30 06:02:29.392659 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392607 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-podinfo\") pod \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " Oct 30 06:02:29.392888 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392710 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-certs" pod "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" (UID: "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.392888 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392776 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-token\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-token\") pod \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " Oct 30 06:02:29.392999 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392931 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-credential-socket" pod "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" (UID: "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.392999 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392941 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-envoy\") pod \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " Oct 30 06:02:29.392999 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.392962 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istiod-ca-cert" pod "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" (UID: "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:29.393235 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.393034 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-socket\") pod \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\" (UID: \"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d\") " Oct 30 06:02:29.393235 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.393124 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-data" pod "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" (UID: "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.396010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.395711 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-socket" pod "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" (UID: "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.396218 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.396129 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.396218 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.396182 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.396218 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.396196 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.396218 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.396209 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.401800 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.396242 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.510870 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.475693 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-token" pod "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" (UID: "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:29.510870 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.493475 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-podinfo" pod "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" (UID: "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:02:29.510870 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.504116 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-envoy" pod "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" (UID: "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.510870 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.506716 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-kube-api-access-44vvx" pod "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" (UID: "5f6da94b-aa21-4e3b-9219-35fac0bc7f3d"). InnerVolumeSpecName "kube-api-access-44vvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:29.511577 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.511543 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.511577 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.511577 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.613420 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.613383 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44vvx\" (UniqueName: \"kubernetes.io/projected/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-kube-api-access-44vvx\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.613420 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.613419 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.749032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.748997 5072 generic.go:356] "Generic (PLEG): container finished" podID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerID="6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08" exitCode=0 Oct 30 06:02:29.749200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.749046 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" Oct 30 06:02:29.749200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.749078 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" event={"ID":"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d","Type":"ContainerDied","Data":"6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08"} Oct 30 06:02:29.749200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.749138 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l" event={"ID":"5f6da94b-aa21-4e3b-9219-35fac0bc7f3d","Type":"ContainerDied","Data":"0eb53270428d56a7f9786698573891642e62928b428e46ecd3d401d7da32efe1"} Oct 30 06:02:29.749200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.749162 5072 scope.go:122] "RemoveContainer" containerID="6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08" Oct 30 06:02:29.762844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.762826 5072 scope.go:122] "RemoveContainer" containerID="6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08" Oct 30 06:02:29.763206 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:02:29.763181 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08\": container with ID starting with 6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08 not found: ID does not exist" containerID="6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08" Oct 30 06:02:29.763312 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.763214 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08"} err="failed to get container status \"6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08\": rpc error: code = NotFound desc = could not find container \"6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08\": container with ID starting with 6c271d013c22fcded73a766d8d64ce3ba20659cef4046fb368a43b9e6b961e08 not found: ID does not exist" Oct 30 06:02:29.804709 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.804678 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l"] Oct 30 06:02:29.816142 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:29.816114 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-np52l"] Oct 30 06:02:30.524992 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:30.524961 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" path="/var/lib/kubelet/pods/5f6da94b-aa21-4e3b-9219-35fac0bc7f3d/volumes" Oct 30 06:02:30.713682 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:02:30.713646 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:02:31.031700 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.031667 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v"] Oct 30 06:02:31.033057 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.033035 5072 cpu_manager.go:414] "containerMap: removing container" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" Oct 30 06:02:31.033110 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.033061 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" Oct 30 06:02:31.033309 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.033296 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6da94b-aa21-4e3b-9219-35fac0bc7f3d" containerName="istio-proxy" Oct 30 06:02:31.039842 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.039821 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.061518 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.056208 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v"] Oct 30 06:02:31.061665 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.061593 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-38cfad-serving-cert\"" Oct 30 06:02:31.139494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.139441 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8bt8\" (UniqueName: \"kubernetes.io/projected/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-kube-api-access-w8bt8\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.139661 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.139552 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-metrics-certs\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.139747 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.139718 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.139785 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.139772 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-openshift-service-ca\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.264484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.254918 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8bt8\" (UniqueName: \"kubernetes.io/projected/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-kube-api-access-w8bt8\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.264484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.255020 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-metrics-certs\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.264484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.255138 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.264484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.255173 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-openshift-service-ca\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.264876 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:02:31.264504 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-serving-cert: secret "maas-api-e2e-inf-b-38cfad-serving-cert" not found Oct 30 06:02:31.264876 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:02:31.264584 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls podName:6cb02ddc-9823-4971-aaa4-b9d98cf4d857 nodeName:}" failed. No retries permitted until 2026-10-30 06:02:31.764565509 +0000 UTC m=+1911.368254747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls") pod "maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" (UID: "6cb02ddc-9823-4971-aaa4-b9d98cf4d857") : secret "maas-api-e2e-inf-b-38cfad-serving-cert" not found Oct 30 06:02:31.271482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.269284 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-openshift-service-ca\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.319527 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.319083 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-metrics-certs\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.366367 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.366331 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8bt8\" (UniqueName: \"kubernetes.io/projected/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-kube-api-access-w8bt8\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.771576 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.771544 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.774227 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.774201 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls\") pod \"maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:31.991991 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:31.991958 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:32.305874 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:02:32.305815 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb02ddc_9823_4971_aaa4_b9d98cf4d857.slice/crio-cbf35b22811a9861075ab4a602ec255c2c08c9206c054a165e5b56c74669bdae WatchSource:0}: Error finding container cbf35b22811a9861075ab4a602ec255c2c08c9206c054a165e5b56c74669bdae: Status 404 returned error can't find the container with id cbf35b22811a9861075ab4a602ec255c2c08c9206c054a165e5b56c74669bdae Oct 30 06:02:32.307202 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:32.307175 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v"] Oct 30 06:02:32.782636 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:32.782594 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" event={"ID":"6cb02ddc-9823-4971-aaa4-b9d98cf4d857","Type":"ContainerStarted","Data":"cbf35b22811a9861075ab4a602ec255c2c08c9206c054a165e5b56c74669bdae"} Oct 30 06:02:33.792941 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:33.792904 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" event={"ID":"6cb02ddc-9823-4971-aaa4-b9d98cf4d857","Type":"ContainerStarted","Data":"881b733658374f0c68bd6742af1b63f6da43cade9e891ea3f84d516ab68c6d6e"} Oct 30 06:02:33.793335 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:33.792981 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:33.838425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:33.838366 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" podStartSLOduration=2.38690838 podStartE2EDuration="2.838347426s" totalImagesPullingTime="451.439046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:33.834338962 +0000 UTC m=+1913.438028220" watchObservedRunningTime="2026-10-30 06:02:33.838347426 +0000 UTC m=+1913.442036684" Oct 30 06:02:36.161346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:36.161307 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd"] Oct 30 06:02:39.806214 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:39.806184 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:02:41.167656 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:41.167601 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" podUID="20c94c6b-a6cf-4737-87c3-909cba482a5a" containerName="payload-pre-processing" containerID="cri-o://ce74df911b4a8f470e21b1768b83b4d385560c87e279bb14273b2d6051e960ff" gracePeriod=25 Oct 30 06:02:48.532950 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:48.532908 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" podUID="20c94c6b-a6cf-4737-87c3-909cba482a5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:02:51.735485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:51.728358 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:58.532343 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:02:58.532301 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" podUID="20c94c6b-a6cf-4737-87c3-909cba482a5a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:03:00.712819 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:03:00.712779 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:03:01.379849 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:01.379815 5072 generic.go:356] "Generic (PLEG): container finished" podID="20c94c6b-a6cf-4737-87c3-909cba482a5a" containerID="ce74df911b4a8f470e21b1768b83b4d385560c87e279bb14273b2d6051e960ff" exitCode=0 Oct 30 06:03:01.380069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:01.379994 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" event={"ID":"20c94c6b-a6cf-4737-87c3-909cba482a5a","Type":"ContainerDied","Data":"ce74df911b4a8f470e21b1768b83b4d385560c87e279bb14273b2d6051e960ff"} Oct 30 06:03:01.418271 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:01.418240 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 06:03:01.554723 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:01.554598 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c94c6b-a6cf-4737-87c3-909cba482a5a-kube-api-access-dkpsv\" (UniqueName: \"kubernetes.io/projected/20c94c6b-a6cf-4737-87c3-909cba482a5a-kube-api-access-dkpsv\") pod \"20c94c6b-a6cf-4737-87c3-909cba482a5a\" (UID: \"20c94c6b-a6cf-4737-87c3-909cba482a5a\") " Oct 30 06:03:01.555056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:01.555035 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c94c6b-a6cf-4737-87c3-909cba482a5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c94c6b-a6cf-4737-87c3-909cba482a5a-plugins-config-volume\") pod \"20c94c6b-a6cf-4737-87c3-909cba482a5a\" (UID: \"20c94c6b-a6cf-4737-87c3-909cba482a5a\") " Oct 30 06:03:01.555715 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:01.555672 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c94c6b-a6cf-4737-87c3-909cba482a5a-plugins-config-volume" pod "20c94c6b-a6cf-4737-87c3-909cba482a5a" (UID: "20c94c6b-a6cf-4737-87c3-909cba482a5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:01.556201 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:01.556135 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c94c6b-a6cf-4737-87c3-909cba482a5a-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:01.558178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:01.558149 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c94c6b-a6cf-4737-87c3-909cba482a5a-kube-api-access-dkpsv" pod "20c94c6b-a6cf-4737-87c3-909cba482a5a" (UID: "20c94c6b-a6cf-4737-87c3-909cba482a5a"). InnerVolumeSpecName "kube-api-access-dkpsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:01.657608 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:01.657577 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkpsv\" (UniqueName: \"kubernetes.io/projected/20c94c6b-a6cf-4737-87c3-909cba482a5a-kube-api-access-dkpsv\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:02.389282 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:02.389249 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" event={"ID":"20c94c6b-a6cf-4737-87c3-909cba482a5a","Type":"ContainerDied","Data":"17640f11d9a9f3e8250f35cb464caea7dce8590d78df1e694cbc867c63bc68da"} Oct 30 06:03:02.389771 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:02.389311 5072 scope.go:122] "RemoveContainer" containerID="ce74df911b4a8f470e21b1768b83b4d385560c87e279bb14273b2d6051e960ff" Oct 30 06:03:02.389771 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:02.389261 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd" Oct 30 06:03:02.462718 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:02.462688 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd"] Oct 30 06:03:02.474776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:02.474750 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39147c-5bc6d8f7c5-khhtd"] Oct 30 06:03:02.524221 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:02.524184 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c94c6b-a6cf-4737-87c3-909cba482a5a" path="/var/lib/kubelet/pods/20c94c6b-a6cf-4737-87c3-909cba482a5a/volumes" Oct 30 06:03:30.711879 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:03:30.711835 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:03:42.158516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:42.156942 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm"] Oct 30 06:03:42.158516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:42.157676 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" containerID="cri-o://9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd" gracePeriod=30 Oct 30 06:03:45.876706 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:45.876670 5072 patch_prober.go:29] interesting pod/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:03:45.877135 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:45.876732 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:03:46.879670 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:46.879631 5072 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:03:46.880244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:46.879714 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:47.547883 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.547851 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 06:03:47.645995 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.645958 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-podinfo\") pod \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " Oct 30 06:03:47.646161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.646016 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-credential-socket\") pod \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " Oct 30 06:03:47.646161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.646076 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-certs\") pod \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " Oct 30 06:03:47.646253 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.646174 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-data\") pod \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " Oct 30 06:03:47.646291 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.646279 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-envoy\") pod \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " Oct 30 06:03:47.646333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.646322 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-token\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-token\") pod \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " Oct 30 06:03:47.646407 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.646392 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-kube-api-access-r5cb7\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-kube-api-access-r5cb7\") pod \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " Oct 30 06:03:47.646448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.646425 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-socket\") pod \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " Oct 30 06:03:47.646550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.646534 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istiod-ca-cert\") pod \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\" (UID: \"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9\") " Oct 30 06:03:47.648012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.647975 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istiod-ca-cert" pod "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" (UID: "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:47.650839 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.650811 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-podinfo" pod "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" (UID: "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:03:47.651034 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.651012 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-credential-socket" pod "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" (UID: "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:47.651210 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.651185 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-certs" pod "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" (UID: "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:47.651387 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.651364 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-data" pod "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" (UID: "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:47.652971 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.652154 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-socket" pod "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" (UID: "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:47.656556 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.656519 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-token" pod "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" (UID: "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:47.656862 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.656756 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-envoy" pod "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" (UID: "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:47.656862 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.656784 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-kube-api-access-r5cb7" pod "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" (UID: "04d5dbc8-b536-4adf-adb7-fab4be7cd1c9"). InnerVolumeSpecName "kube-api-access-r5cb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:47.749167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.749067 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:47.749167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.749104 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:47.749167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.749117 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:47.749167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.749130 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5cb7\" (UniqueName: \"kubernetes.io/projected/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-kube-api-access-r5cb7\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:47.749167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.749143 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:47.749167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.749157 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:47.749167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.749171 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:47.749167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.749182 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:47.749689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:47.749195 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:03:48.217387 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.217351 5072 generic.go:356] "Generic (PLEG): container finished" podID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerID="9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd" exitCode=0 Oct 30 06:03:48.217854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.217406 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" Oct 30 06:03:48.217854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.217430 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" event={"ID":"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9","Type":"ContainerDied","Data":"9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd"} Oct 30 06:03:48.217854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.217510 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm" event={"ID":"04d5dbc8-b536-4adf-adb7-fab4be7cd1c9","Type":"ContainerDied","Data":"88313720736d54ba6b9da1947f5f274a379ad408bee6c47053b18acfdfe0a298"} Oct 30 06:03:48.217854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.217543 5072 scope.go:122] "RemoveContainer" containerID="9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd" Oct 30 06:03:48.233728 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.233702 5072 scope.go:122] "RemoveContainer" containerID="04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006" Oct 30 06:03:48.244629 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.244605 5072 scope.go:122] "RemoveContainer" containerID="9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd" Oct 30 06:03:48.244928 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:03:48.244905 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd\": container with ID starting with 9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd not found: ID does not exist" containerID="9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd" Oct 30 06:03:48.245018 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.244937 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd"} err="failed to get container status \"9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd\": rpc error: code = NotFound desc = could not find container \"9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd\": container with ID starting with 9a408e0d8121f0de80f12832c015b242464643d6d1cc93656ffa7dde644257fd not found: ID does not exist" Oct 30 06:03:48.245018 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.244958 5072 scope.go:122] "RemoveContainer" containerID="04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006" Oct 30 06:03:48.245280 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:03:48.245258 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006\": container with ID starting with 04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006 not found: ID does not exist" containerID="04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006" Oct 30 06:03:48.245338 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.245288 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006"} err="failed to get container status \"04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006\": rpc error: code = NotFound desc = could not find container \"04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006\": container with ID starting with 04ec5a87df2bbeadb0d94468fe30d73203d04a9c6b43ad7053cf4644610c8006 not found: ID does not exist" Oct 30 06:03:48.298577 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.298546 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm"] Oct 30 06:03:48.310795 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.310765 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-39147c-openshift-default-74f45c8f87-s7jpm"] Oct 30 06:03:48.522376 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:48.522298 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" path="/var/lib/kubelet/pods/04d5dbc8-b536-4adf-adb7-fab4be7cd1c9/volumes" Oct 30 06:03:50.240675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:50.240590 5072 generic.go:356] "Generic (PLEG): container finished" podID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerID="32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96" exitCode=0 Oct 30 06:03:50.241090 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:50.240671 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" event={"ID":"e4e2d554-5320-439f-a7cb-133f6c6e4780","Type":"ContainerDied","Data":"32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96"} Oct 30 06:03:50.241177 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:50.241159 5072 scope.go:122] "RemoveContainer" containerID="32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96" Oct 30 06:03:50.243269 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:50.243238 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:03:50.243376 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:50.243305 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:03:50.243376 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:50.243338 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:03:51.267489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:51.267434 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" event={"ID":"e4e2d554-5320-439f-a7cb-133f6c6e4780","Type":"ContainerStarted","Data":"8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0"} Oct 30 06:03:51.768644 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:51.768603 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:03:51.768878 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:51.768856 5072 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 06:03:51.768965 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:51.768944 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 06:03:52.769400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:52.769367 5072 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 06:03:52.769808 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:52.769420 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 06:03:53.769259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:53.769224 5072 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 06:03:53.769437 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:53.769280 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 06:03:54.769389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:54.769356 5072 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 06:03:54.769609 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:54.769414 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 06:03:55.769624 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:55.769588 5072 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 06:03:55.770014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:55.769638 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 06:03:56.768675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:56.768636 5072 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 06:03:56.768862 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:56.768699 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 06:03:57.774732 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:57.774701 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:03:57.775227 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:57.775206 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:03:57.776002 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:03:57.775984 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:04:00.713707 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:04:00.713662 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:04:04.466325 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.464120 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj"] Oct 30 06:04:04.470268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.467942 5072 cpu_manager.go:414] "containerMap: removing container" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" Oct 30 06:04:04.470268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.467971 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" Oct 30 06:04:04.470268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.467991 5072 cpu_manager.go:414] "containerMap: removing container" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" Oct 30 06:04:04.470268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.467999 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" Oct 30 06:04:04.470268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.468075 5072 cpu_manager.go:414] "containerMap: removing container" podUID="20c94c6b-a6cf-4737-87c3-909cba482a5a" containerName="payload-pre-processing" Oct 30 06:04:04.470268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.468083 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c94c6b-a6cf-4737-87c3-909cba482a5a" containerName="payload-pre-processing" Oct 30 06:04:04.470268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.468352 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c94c6b-a6cf-4737-87c3-909cba482a5a" containerName="payload-pre-processing" Oct 30 06:04:04.470268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.468382 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" Oct 30 06:04:04.470268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.468394 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d5dbc8-b536-4adf-adb7-fab4be7cd1c9" containerName="istio-proxy" Oct 30 06:04:04.474190 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.474165 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.487801 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.487774 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj"] Oct 30 06:04:04.492622 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.492606 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-482468\"/\"default-dockercfg-4lzbx\"" Oct 30 06:04:04.492823 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.492811 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-482468\"/\"test-model-482468-kserve-self-signed-certs\"" Oct 30 06:04:04.587066 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.587028 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-482468\"/\"kube-root-ca.crt\"" Oct 30 06:04:04.588789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.588760 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-482468\"/\"openshift-service-ca.crt\"" Oct 30 06:04:04.617014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.616984 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tls-certs\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.617170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.617037 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-home\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.617170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.617135 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tmp-dir\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.617291 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.617265 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqzq5\" (UniqueName: \"kubernetes.io/projected/283adf34-7bce-4a42-9a6a-fbd8009ca69a-kube-api-access-mqzq5\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.617333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.617307 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-model-cache\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.617567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.617548 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-dshm\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.719010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.718918 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-dshm\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.719185 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.719101 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tls-certs\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.719185 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.719157 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-home\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.719305 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.719201 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tmp-dir\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.719305 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.719275 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqzq5\" (UniqueName: \"kubernetes.io/projected/283adf34-7bce-4a42-9a6a-fbd8009ca69a-kube-api-access-mqzq5\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.719416 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.719303 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-model-cache\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.719624 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.719596 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-home\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.719780 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.719753 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-model-cache\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.720019 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.719994 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tmp-dir\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.721528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.721503 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-dshm\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.722211 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.722191 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tls-certs\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.780482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.773177 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqzq5\" (UniqueName: \"kubernetes.io/projected/283adf34-7bce-4a42-9a6a-fbd8009ca69a-kube-api-access-mqzq5\") pod \"test-model-482468-kserve-6959c6c555-zfxnj\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:04.799570 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:04.798274 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:05.298056 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:04:05.298024 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod283adf34_7bce_4a42_9a6a_fbd8009ca69a.slice/crio-c5798ed2459daafa420976a382cbe5c99f442ec761633ac24a8ed57b5e149f96 WatchSource:0}: Error finding container c5798ed2459daafa420976a382cbe5c99f442ec761633ac24a8ed57b5e149f96: Status 404 returned error can't find the container with id c5798ed2459daafa420976a382cbe5c99f442ec761633ac24a8ed57b5e149f96 Oct 30 06:04:05.298970 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:05.298947 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj"] Oct 30 06:04:05.486769 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:05.486727 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" event={"ID":"283adf34-7bce-4a42-9a6a-fbd8009ca69a","Type":"ContainerStarted","Data":"e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7"} Oct 30 06:04:05.486769 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:05.486779 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" event={"ID":"283adf34-7bce-4a42-9a6a-fbd8009ca69a","Type":"ContainerStarted","Data":"c5798ed2459daafa420976a382cbe5c99f442ec761633ac24a8ed57b5e149f96"} Oct 30 06:04:05.535032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:05.534981 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" podStartSLOduration=1.534966078 podStartE2EDuration="1.534966078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:05.532376495 +0000 UTC m=+2005.136065753" watchObservedRunningTime="2026-10-30 06:04:05.534966078 +0000 UTC m=+2005.138655334" Oct 30 06:04:05.799223 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:05.799192 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:14.798943 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:14.798902 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:14.814114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:14.814083 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:15.650143 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:15.650103 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:04:21.318861 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.318810 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb"] Oct 30 06:04:21.326255 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.326230 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.347015 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.346987 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb"] Oct 30 06:04:21.349854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.349831 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f28e2d\"/\"default-dockercfg-gw7rf\"" Oct 30 06:04:21.350006 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.349903 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f28e2d\"/\"test-model-f28e2d-kserve-self-signed-certs\"" Oct 30 06:04:21.364036 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.356997 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f28e2d\"/\"openshift-service-ca.crt\"" Oct 30 06:04:21.364036 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.357109 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f28e2d\"/\"kube-root-ca.crt\"" Oct 30 06:04:21.367374 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.367345 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-home\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.367564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.367499 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-dshm\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.367564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.367556 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlnh\" (UniqueName: \"kubernetes.io/projected/ad9e7a96-c07a-4740-9e83-8e2179fff394-kube-api-access-stlnh\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.367712 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.367664 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9e7a96-c07a-4740-9e83-8e2179fff394-tls-certs\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.367780 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.367711 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-tmp-dir\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.367845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.367827 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-model-cache\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.469178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.469138 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-home\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.469386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.469219 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-dshm\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.469386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.469257 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlnh\" (UniqueName: \"kubernetes.io/projected/ad9e7a96-c07a-4740-9e83-8e2179fff394-kube-api-access-stlnh\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.469386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.469289 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9e7a96-c07a-4740-9e83-8e2179fff394-tls-certs\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.469386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.469310 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-tmp-dir\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.469386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.469383 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-model-cache\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.469729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.469617 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-home\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.469827 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.469805 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-tmp-dir\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.469902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.469851 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-model-cache\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.471854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.471831 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-dshm\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.472619 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.472598 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9e7a96-c07a-4740-9e83-8e2179fff394-tls-certs\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.492470 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.492427 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlnh\" (UniqueName: \"kubernetes.io/projected/ad9e7a96-c07a-4740-9e83-8e2179fff394-kube-api-access-stlnh\") pod \"test-model-f28e2d-kserve-6569db5d77-nptbb\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:21.659485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:21.658776 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:22.106586 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:04:22.106555 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad9e7a96_c07a_4740_9e83_8e2179fff394.slice/crio-bb253a8eb6b958881dc1151568384a7e69e539589049fc811fa0b2c7774a8680 WatchSource:0}: Error finding container bb253a8eb6b958881dc1151568384a7e69e539589049fc811fa0b2c7774a8680: Status 404 returned error can't find the container with id bb253a8eb6b958881dc1151568384a7e69e539589049fc811fa0b2c7774a8680 Oct 30 06:04:22.108477 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:22.108432 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb"] Oct 30 06:04:22.760422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:22.760385 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" event={"ID":"ad9e7a96-c07a-4740-9e83-8e2179fff394","Type":"ContainerStarted","Data":"b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a"} Oct 30 06:04:22.760897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:22.760437 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" event={"ID":"ad9e7a96-c07a-4740-9e83-8e2179fff394","Type":"ContainerStarted","Data":"bb253a8eb6b958881dc1151568384a7e69e539589049fc811fa0b2c7774a8680"} Oct 30 06:04:22.803264 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:22.803212 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" podStartSLOduration=1.8031969920000002 podStartE2EDuration="1.803196992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:22.801385081 +0000 UTC m=+2022.405074334" watchObservedRunningTime="2026-10-30 06:04:22.803196992 +0000 UTC m=+2022.406886248" Oct 30 06:04:23.659607 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:23.659576 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:29.999770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:29.999732 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg"] Oct 30 06:04:30.006568 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.006537 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.029223 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.029192 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg"] Oct 30 06:04:30.032079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.032053 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7dfcb7\"/\"test-model-7dfcb7-kserve-self-signed-certs\"" Oct 30 06:04:30.037068 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.037048 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7dfcb7\"/\"default-dockercfg-x2wnh\"" Oct 30 06:04:30.037068 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.037048 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7dfcb7\"/\"openshift-service-ca.crt\"" Oct 30 06:04:30.037068 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.037070 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7dfcb7\"/\"kube-root-ca.crt\"" Oct 30 06:04:30.127899 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.127867 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-model-cache\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.128106 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.127944 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tls-certs\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.128106 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.128089 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-home\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.128252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.128133 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-dshm\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.128417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.128394 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tmp-dir\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.128543 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.128519 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5jqp\" (UniqueName: \"kubernetes.io/projected/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-kube-api-access-n5jqp\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.255102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.251734 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tmp-dir\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.255102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.251798 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5jqp\" (UniqueName: \"kubernetes.io/projected/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-kube-api-access-n5jqp\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.255102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.251942 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-model-cache\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.255102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.252027 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tls-certs\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.255102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.252100 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-home\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.255102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.252132 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-dshm\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.257932 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.255982 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-model-cache\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.257932 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.256766 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tmp-dir\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.261495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.259793 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-home\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.284482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.280604 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-dshm\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.286813 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.286785 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5jqp\" (UniqueName: \"kubernetes.io/projected/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-kube-api-access-n5jqp\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.315805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.313408 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tls-certs\") pod \"test-model-7dfcb7-kserve-7945958485-8hlwg\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.328593 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.328564 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:30.712388 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:04:30.712341 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:04:30.816444 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.816415 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg"] Oct 30 06:04:30.816930 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:04:30.816907 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d6811e9_6b55_4ec8_bccd_a34c011fe27e.slice/crio-425b87fa513b8fe968b3eedf1697a660066fbab279689019e6288d97765eeb5b WatchSource:0}: Error finding container 425b87fa513b8fe968b3eedf1697a660066fbab279689019e6288d97765eeb5b: Status 404 returned error can't find the container with id 425b87fa513b8fe968b3eedf1697a660066fbab279689019e6288d97765eeb5b Oct 30 06:04:30.839970 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:30.839933 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" event={"ID":"2d6811e9-6b55-4ec8-bccd-a34c011fe27e","Type":"ContainerStarted","Data":"425b87fa513b8fe968b3eedf1697a660066fbab279689019e6288d97765eeb5b"} Oct 30 06:04:31.660079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:31.660046 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:31.673594 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:31.673568 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:31.850502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:31.850445 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" event={"ID":"2d6811e9-6b55-4ec8-bccd-a34c011fe27e","Type":"ContainerStarted","Data":"78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345"} Oct 30 06:04:31.862526 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:31.862495 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:04:31.896162 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:31.896113 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" podStartSLOduration=2.896099552 podStartE2EDuration="2.896099552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:31.891526397 +0000 UTC m=+2031.495215666" watchObservedRunningTime="2026-10-30 06:04:31.896099552 +0000 UTC m=+2031.499788808" Oct 30 06:04:32.329897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:32.329863 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:36.009259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.009224 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56"] Oct 30 06:04:36.015364 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.015322 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.045555 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.045447 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-02ceed\"/\"default-dockercfg-s5wrt\"" Oct 30 06:04:36.045737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.045452 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-02ceed\"/\"kube-root-ca.crt\"" Oct 30 06:04:36.045737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.045630 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-02ceed\"/\"auth-test-model-02ceed-kserve-self-signed-certs\"" Oct 30 06:04:36.045737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.045677 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-02ceed\"/\"openshift-service-ca.crt\"" Oct 30 06:04:36.051266 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.051240 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56"] Oct 30 06:04:36.098107 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.098071 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tmp-dir\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.098309 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.098128 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-dshm\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.098470 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.098425 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-home\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.098584 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.098563 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-model-cache\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.098660 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.098614 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tls-certs\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.098749 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.098728 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsb7p\" (UniqueName: \"kubernetes.io/projected/7ee09dc1-83a8-40af-9a26-7ac225f8506d-kube-api-access-xsb7p\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.200608 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.200569 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tmp-dir\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.200608 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.200624 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-dshm\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.200873 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.200789 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-home\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.200873 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.200834 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-model-cache\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.200981 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.200874 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tls-certs\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.200981 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.200945 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsb7p\" (UniqueName: \"kubernetes.io/projected/7ee09dc1-83a8-40af-9a26-7ac225f8506d-kube-api-access-xsb7p\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.202071 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.202042 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tmp-dir\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.244604 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.216868 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-home\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.244933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.244308 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-model-cache\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.301034 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.300916 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tls-certs\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.305012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.304941 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-dshm\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.315485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.308161 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsb7p\" (UniqueName: \"kubernetes.io/projected/7ee09dc1-83a8-40af-9a26-7ac225f8506d-kube-api-access-xsb7p\") pod \"auth-test-model-02ceed-kserve-686fccf5fc-f9h56\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.348511 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.339768 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:36.790285 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.790255 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56"] Oct 30 06:04:36.911016 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.910976 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" event={"ID":"7ee09dc1-83a8-40af-9a26-7ac225f8506d","Type":"ContainerStarted","Data":"35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204"} Oct 30 06:04:36.911222 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.911043 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" event={"ID":"7ee09dc1-83a8-40af-9a26-7ac225f8506d","Type":"ContainerStarted","Data":"a31ae4d1f6229f7f76cc1f4472ccc381f47e03cf8f76f47365ef085a0044ca1d"} Oct 30 06:04:36.964392 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:36.964327 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" podStartSLOduration=1.96430975 podStartE2EDuration="1.96430975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:36.960387507 +0000 UTC m=+2036.564076765" watchObservedRunningTime="2026-10-30 06:04:36.96430975 +0000 UTC m=+2036.567999011" Oct 30 06:04:37.341085 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:37.341055 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:40.329750 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.329707 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:40.344088 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.344058 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:40.446497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.446451 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm"] Oct 30 06:04:40.456665 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.456634 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.477969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.477921 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-080391-serving-cert\"" Oct 30 06:04:40.481978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.481932 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm"] Oct 30 06:04:40.490240 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.490203 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.490545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.490521 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bad623d-f3f5-4d28-be15-7512312eb5c6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.490726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.490699 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.491385 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.490955 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmjjz\" (UniqueName: \"kubernetes.io/projected/1bad623d-f3f5-4d28-be15-7512312eb5c6-kube-api-access-kmjjz\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.592211 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.592126 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmjjz\" (UniqueName: \"kubernetes.io/projected/1bad623d-f3f5-4d28-be15-7512312eb5c6-kube-api-access-kmjjz\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.592369 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.592288 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.592369 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.592359 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bad623d-f3f5-4d28-be15-7512312eb5c6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.592505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.592390 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.592902 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:04:40.592886 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-serving-cert: secret "maas-api-e2e-shared-a-w3-080391-serving-cert" not found Oct 30 06:04:40.592949 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:04:40.592941 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls podName:1bad623d-f3f5-4d28-be15-7512312eb5c6 nodeName:}" failed. No retries permitted until 2026-10-30 06:04:41.092926983 +0000 UTC m=+2040.696616219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls") pod "maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" (UID: "1bad623d-f3f5-4d28-be15-7512312eb5c6") : secret "maas-api-e2e-shared-a-w3-080391-serving-cert" not found Oct 30 06:04:40.593578 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.593552 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bad623d-f3f5-4d28-be15-7512312eb5c6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.595529 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.595503 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.618494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.618423 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmjjz\" (UniqueName: \"kubernetes.io/projected/1bad623d-f3f5-4d28-be15-7512312eb5c6-kube-api-access-kmjjz\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:40.956760 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:40.956733 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:04:41.129302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:41.129187 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:41.134117 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:41.134090 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:41.374983 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:41.374952 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:41.759426 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:04:41.759383 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bad623d_f3f5_4d28_be15_7512312eb5c6.slice/crio-994f649a7891283592dd9a983016cd70eb420966b16ec4bac9bb160574cfd664 WatchSource:0}: Error finding container 994f649a7891283592dd9a983016cd70eb420966b16ec4bac9bb160574cfd664: Status 404 returned error can't find the container with id 994f649a7891283592dd9a983016cd70eb420966b16ec4bac9bb160574cfd664 Oct 30 06:04:41.764107 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:41.764065 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm"] Oct 30 06:04:42.017731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:42.017641 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" event={"ID":"1bad623d-f3f5-4d28-be15-7512312eb5c6","Type":"ContainerStarted","Data":"994f649a7891283592dd9a983016cd70eb420966b16ec4bac9bb160574cfd664"} Oct 30 06:04:43.028346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:43.028309 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" event={"ID":"1bad623d-f3f5-4d28-be15-7512312eb5c6","Type":"ContainerStarted","Data":"690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f"} Oct 30 06:04:43.028869 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:43.028370 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:04:46.340537 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:46.340501 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:46.353968 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:46.353937 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:46.400867 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:46.400822 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" podStartSLOduration=5.872185771 podStartE2EDuration="6.400808588s" totalImagesPullingTime="528.622817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:43.092909557 +0000 UTC m=+2042.696598814" watchObservedRunningTime="2026-10-30 06:04:46.400808588 +0000 UTC m=+2046.004497845" Oct 30 06:04:47.078019 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:47.077992 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:04:49.045302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:04:49.045268 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:05:00.711739 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:05:00.711693 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:05:02.341493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:02.335580 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:02.656574 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:02.647124 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" podUID="283adf34-7bce-4a42-9a6a-fbd8009ca69a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.168:8000/ready\": context deadline exceeded" Oct 30 06:05:05.709351 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:05.709311 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz"] Oct 30 06:05:05.709900 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:05.709570 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" containerID="cri-o://631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651" gracePeriod=30 Oct 30 06:05:11.102319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.102293 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 06:05:11.128214 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128170 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-data\") pod \"fb610158-0784-4321-abee-9f48c8ba2bba\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " Oct 30 06:05:11.128410 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128271 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-socket\") pod \"fb610158-0784-4321-abee-9f48c8ba2bba\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " Oct 30 06:05:11.128410 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128300 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-credential-socket\") pod \"fb610158-0784-4321-abee-9f48c8ba2bba\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " Oct 30 06:05:11.128410 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128369 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb610158-0784-4321-abee-9f48c8ba2bba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb610158-0784-4321-abee-9f48c8ba2bba-istio-podinfo\") pod \"fb610158-0784-4321-abee-9f48c8ba2bba\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " Oct 30 06:05:11.128410 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128408 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb610158-0784-4321-abee-9f48c8ba2bba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb610158-0784-4321-abee-9f48c8ba2bba-istiod-ca-cert\") pod \"fb610158-0784-4321-abee-9f48c8ba2bba\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " Oct 30 06:05:11.128788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128511 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-certs\") pod \"fb610158-0784-4321-abee-9f48c8ba2bba\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " Oct 30 06:05:11.128788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128514 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-data" pod "fb610158-0784-4321-abee-9f48c8ba2bba" (UID: "fb610158-0784-4321-abee-9f48c8ba2bba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.128788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128545 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-envoy\") pod \"fb610158-0784-4321-abee-9f48c8ba2bba\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " Oct 30 06:05:11.128788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128588 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-socket" pod "fb610158-0784-4321-abee-9f48c8ba2bba" (UID: "fb610158-0784-4321-abee-9f48c8ba2bba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.128788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128652 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-kube-api-access-lrntx\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-kube-api-access-lrntx\") pod \"fb610158-0784-4321-abee-9f48c8ba2bba\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " Oct 30 06:05:11.128788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128774 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-istio-token\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-istio-token\") pod \"fb610158-0784-4321-abee-9f48c8ba2bba\" (UID: \"fb610158-0784-4321-abee-9f48c8ba2bba\") " Oct 30 06:05:11.129124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128654 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-credential-socket" pod "fb610158-0784-4321-abee-9f48c8ba2bba" (UID: "fb610158-0784-4321-abee-9f48c8ba2bba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.129124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.128952 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb610158-0784-4321-abee-9f48c8ba2bba-istiod-ca-cert" pod "fb610158-0784-4321-abee-9f48c8ba2bba" (UID: "fb610158-0784-4321-abee-9f48c8ba2bba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:11.129124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.129083 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-certs" pod "fb610158-0784-4321-abee-9f48c8ba2bba" (UID: "fb610158-0784-4321-abee-9f48c8ba2bba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.130877 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.130849 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.131056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.130881 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.131056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.130896 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.131056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.130910 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb610158-0784-4321-abee-9f48c8ba2bba-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.131056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.130924 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.131911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.131883 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-kube-api-access-lrntx" pod "fb610158-0784-4321-abee-9f48c8ba2bba" (UID: "fb610158-0784-4321-abee-9f48c8ba2bba"). InnerVolumeSpecName "kube-api-access-lrntx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:11.132081 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.132065 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-envoy" pod "fb610158-0784-4321-abee-9f48c8ba2bba" (UID: "fb610158-0784-4321-abee-9f48c8ba2bba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.132311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.132288 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb610158-0784-4321-abee-9f48c8ba2bba-istio-podinfo" pod "fb610158-0784-4321-abee-9f48c8ba2bba" (UID: "fb610158-0784-4321-abee-9f48c8ba2bba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:11.132871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.132850 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-istio-token" pod "fb610158-0784-4321-abee-9f48c8ba2bba" (UID: "fb610158-0784-4321-abee-9f48c8ba2bba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:11.233181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.233050 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb610158-0784-4321-abee-9f48c8ba2bba-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.233181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.233085 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb610158-0784-4321-abee-9f48c8ba2bba-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.233181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.233101 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrntx\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-kube-api-access-lrntx\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.233181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.233118 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb610158-0784-4321-abee-9f48c8ba2bba-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.593630 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.593593 5072 generic.go:356] "Generic (PLEG): container finished" podID="fb610158-0784-4321-abee-9f48c8ba2bba" containerID="631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651" exitCode=0 Oct 30 06:05:11.593812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.593642 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" Oct 30 06:05:11.593812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.593674 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" event={"ID":"fb610158-0784-4321-abee-9f48c8ba2bba","Type":"ContainerDied","Data":"631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651"} Oct 30 06:05:11.593812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.593738 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" event={"ID":"fb610158-0784-4321-abee-9f48c8ba2bba","Type":"ContainerDied","Data":"922329c87737ed5d2970804b22819da3d65ddc6e34a04775158786e0d79e0982"} Oct 30 06:05:11.593812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.593766 5072 scope.go:122] "RemoveContainer" containerID="631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651" Oct 30 06:05:11.606585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.606561 5072 scope.go:122] "RemoveContainer" containerID="631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651" Oct 30 06:05:11.606872 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:05:11.606854 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651\": container with ID starting with 631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651 not found: ID does not exist" containerID="631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651" Oct 30 06:05:11.606945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.606881 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651"} err="failed to get container status \"631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651\": rpc error: code = NotFound desc = could not find container \"631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651\": container with ID starting with 631a812a68e943f58917baf3cc634bf360bfc972eb587d88c3b734fef6ccf651 not found: ID does not exist" Oct 30 06:05:11.652242 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.652195 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz"] Oct 30 06:05:11.658536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.658509 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz"] Oct 30 06:05:11.980253 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.980219 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:05:11.980427 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:11.980269 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7zpxz" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:12.523699 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:12.523665 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" path="/var/lib/kubelet/pods/fb610158-0784-4321-abee-9f48c8ba2bba/volumes" Oct 30 06:05:20.588938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:20.588895 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t"] Oct 30 06:05:20.589487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:20.589164 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" containerID="cri-o://e6a5ae7b68936593dbfddbddea1f8b06c10b4ec5d8c7608a532f7beeec4afe1d" gracePeriod=30 Oct 30 06:05:20.995882 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:20.995845 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4"] Oct 30 06:05:20.996099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:20.996074 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" containerID="cri-o://f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468" gracePeriod=30 Oct 30 06:05:22.251207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:22.247010 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g"] Oct 30 06:05:22.251207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:22.247301 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" containerID="cri-o://a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1" gracePeriod=30 Oct 30 06:05:22.868547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:22.868102 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zdt2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:05:22.868547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:22.868156 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:05:24.107874 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:24.107839 5072 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:05:24.108281 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:24.107902 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:05:25.880349 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.880313 5072 generic.go:356] "Generic (PLEG): container finished" podID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerID="e6a5ae7b68936593dbfddbddea1f8b06c10b4ec5d8c7608a532f7beeec4afe1d" exitCode=0 Oct 30 06:05:25.880768 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.880383 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" event={"ID":"80e19c9f-4d40-499d-bf2e-4c92e850391b","Type":"ContainerDied","Data":"e6a5ae7b68936593dbfddbddea1f8b06c10b4ec5d8c7608a532f7beeec4afe1d"} Oct 30 06:05:25.909136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.909107 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 06:05:25.995894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.995858 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-data\") pod \"80e19c9f-4d40-499d-bf2e-4c92e850391b\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " Oct 30 06:05:25.996096 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.995943 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-credential-socket\") pod \"80e19c9f-4d40-499d-bf2e-4c92e850391b\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " Oct 30 06:05:25.996096 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.995978 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-kube-api-access-c5ckj\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-kube-api-access-c5ckj\") pod \"80e19c9f-4d40-499d-bf2e-4c92e850391b\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " Oct 30 06:05:25.996248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.996108 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e19c9f-4d40-499d-bf2e-4c92e850391b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e19c9f-4d40-499d-bf2e-4c92e850391b-istiod-ca-cert\") pod \"80e19c9f-4d40-499d-bf2e-4c92e850391b\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " Oct 30 06:05:25.996248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.996185 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-data" pod "80e19c9f-4d40-499d-bf2e-4c92e850391b" (UID: "80e19c9f-4d40-499d-bf2e-4c92e850391b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:25.996248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.996199 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-certs\") pod \"80e19c9f-4d40-499d-bf2e-4c92e850391b\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " Oct 30 06:05:25.996414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.996273 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-socket\") pod \"80e19c9f-4d40-499d-bf2e-4c92e850391b\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " Oct 30 06:05:25.996414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.996322 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-podinfo\") pod \"80e19c9f-4d40-499d-bf2e-4c92e850391b\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " Oct 30 06:05:25.996560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.996435 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-certs" pod "80e19c9f-4d40-499d-bf2e-4c92e850391b" (UID: "80e19c9f-4d40-499d-bf2e-4c92e850391b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:25.996655 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.996632 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-envoy\") pod \"80e19c9f-4d40-499d-bf2e-4c92e850391b\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " Oct 30 06:05:25.996743 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.996686 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-credential-socket" pod "80e19c9f-4d40-499d-bf2e-4c92e850391b" (UID: "80e19c9f-4d40-499d-bf2e-4c92e850391b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:25.996805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.996759 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-token\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-token\") pod \"80e19c9f-4d40-499d-bf2e-4c92e850391b\" (UID: \"80e19c9f-4d40-499d-bf2e-4c92e850391b\") " Oct 30 06:05:25.997328 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.997290 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e19c9f-4d40-499d-bf2e-4c92e850391b-istiod-ca-cert" pod "80e19c9f-4d40-499d-bf2e-4c92e850391b" (UID: "80e19c9f-4d40-499d-bf2e-4c92e850391b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:25.997689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.997668 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:25.997841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.997826 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:25.997939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.997927 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e19c9f-4d40-499d-bf2e-4c92e850391b-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:25.998049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.998037 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:25.998446 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:25.998420 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-socket" pod "80e19c9f-4d40-499d-bf2e-4c92e850391b" (UID: "80e19c9f-4d40-499d-bf2e-4c92e850391b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.001091 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.001055 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-kube-api-access-c5ckj" pod "80e19c9f-4d40-499d-bf2e-4c92e850391b" (UID: "80e19c9f-4d40-499d-bf2e-4c92e850391b"). InnerVolumeSpecName "kube-api-access-c5ckj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:26.003978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.003922 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-token" pod "80e19c9f-4d40-499d-bf2e-4c92e850391b" (UID: "80e19c9f-4d40-499d-bf2e-4c92e850391b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:26.006975 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.006944 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-podinfo" pod "80e19c9f-4d40-499d-bf2e-4c92e850391b" (UID: "80e19c9f-4d40-499d-bf2e-4c92e850391b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:26.011238 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.011206 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-envoy" pod "80e19c9f-4d40-499d-bf2e-4c92e850391b" (UID: "80e19c9f-4d40-499d-bf2e-4c92e850391b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.100282 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.100250 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.100282 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.100279 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.100282 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.100288 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5ckj\" (UniqueName: \"kubernetes.io/projected/80e19c9f-4d40-499d-bf2e-4c92e850391b-kube-api-access-c5ckj\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.100542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.100298 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e19c9f-4d40-499d-bf2e-4c92e850391b-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.100542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.100307 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e19c9f-4d40-499d-bf2e-4c92e850391b-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.276926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.276894 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 06:05:26.413311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413279 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-socket\") pod \"7710b253-73f0-4ee8-863a-39e6b5e472a1\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " Oct 30 06:05:26.413501 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413383 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-kube-api-access-frfx4\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-kube-api-access-frfx4\") pod \"7710b253-73f0-4ee8-863a-39e6b5e472a1\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " Oct 30 06:05:26.413501 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413423 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-certs\") pod \"7710b253-73f0-4ee8-863a-39e6b5e472a1\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " Oct 30 06:05:26.413661 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413521 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-podinfo\") pod \"7710b253-73f0-4ee8-863a-39e6b5e472a1\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " Oct 30 06:05:26.413661 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413532 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-socket" pod "7710b253-73f0-4ee8-863a-39e6b5e472a1" (UID: "7710b253-73f0-4ee8-863a-39e6b5e472a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.413661 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413562 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-envoy\") pod \"7710b253-73f0-4ee8-863a-39e6b5e472a1\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " Oct 30 06:05:26.413805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413790 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-credential-socket\") pod \"7710b253-73f0-4ee8-863a-39e6b5e472a1\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " Oct 30 06:05:26.413805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413790 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-certs" pod "7710b253-73f0-4ee8-863a-39e6b5e472a1" (UID: "7710b253-73f0-4ee8-863a-39e6b5e472a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.413913 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413837 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7710b253-73f0-4ee8-863a-39e6b5e472a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7710b253-73f0-4ee8-863a-39e6b5e472a1-istiod-ca-cert\") pod \"7710b253-73f0-4ee8-863a-39e6b5e472a1\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " Oct 30 06:05:26.413996 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.413982 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-token\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-token\") pod \"7710b253-73f0-4ee8-863a-39e6b5e472a1\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " Oct 30 06:05:26.414054 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.414029 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-credential-socket" pod "7710b253-73f0-4ee8-863a-39e6b5e472a1" (UID: "7710b253-73f0-4ee8-863a-39e6b5e472a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.414119 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.414102 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-data\") pod \"7710b253-73f0-4ee8-863a-39e6b5e472a1\" (UID: \"7710b253-73f0-4ee8-863a-39e6b5e472a1\") " Oct 30 06:05:26.414912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.414414 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7710b253-73f0-4ee8-863a-39e6b5e472a1-istiod-ca-cert" pod "7710b253-73f0-4ee8-863a-39e6b5e472a1" (UID: "7710b253-73f0-4ee8-863a-39e6b5e472a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:26.414912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.414573 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-data" pod "7710b253-73f0-4ee8-863a-39e6b5e472a1" (UID: "7710b253-73f0-4ee8-863a-39e6b5e472a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.414912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.414856 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.414912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.414869 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7710b253-73f0-4ee8-863a-39e6b5e472a1-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.414912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.414877 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.414912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.414885 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.414912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.414893 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.415965 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.415941 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-envoy" pod "7710b253-73f0-4ee8-863a-39e6b5e472a1" (UID: "7710b253-73f0-4ee8-863a-39e6b5e472a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.416334 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.416317 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-kube-api-access-frfx4" pod "7710b253-73f0-4ee8-863a-39e6b5e472a1" (UID: "7710b253-73f0-4ee8-863a-39e6b5e472a1"). InnerVolumeSpecName "kube-api-access-frfx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:26.416658 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.416631 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-podinfo" pod "7710b253-73f0-4ee8-863a-39e6b5e472a1" (UID: "7710b253-73f0-4ee8-863a-39e6b5e472a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:26.416766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.416673 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-token" pod "7710b253-73f0-4ee8-863a-39e6b5e472a1" (UID: "7710b253-73f0-4ee8-863a-39e6b5e472a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:26.516398 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.516310 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.516398 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.516337 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frfx4\" (UniqueName: \"kubernetes.io/projected/7710b253-73f0-4ee8-863a-39e6b5e472a1-kube-api-access-frfx4\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.516398 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.516346 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.516398 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.516357 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7710b253-73f0-4ee8-863a-39e6b5e472a1-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.891004 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.890971 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" Oct 30 06:05:26.891624 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.890995 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t" event={"ID":"80e19c9f-4d40-499d-bf2e-4c92e850391b","Type":"ContainerDied","Data":"3810cf1e78110e503106a09dbb91199d20dcdc275cdc7ac1cb12ab47bb255f21"} Oct 30 06:05:26.891624 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.891069 5072 scope.go:122] "RemoveContainer" containerID="e6a5ae7b68936593dbfddbddea1f8b06c10b4ec5d8c7608a532f7beeec4afe1d" Oct 30 06:05:26.892987 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.892964 5072 generic.go:356] "Generic (PLEG): container finished" podID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerID="f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468" exitCode=0 Oct 30 06:05:26.893154 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.893009 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" Oct 30 06:05:26.893154 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.893050 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" event={"ID":"7710b253-73f0-4ee8-863a-39e6b5e472a1","Type":"ContainerDied","Data":"f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468"} Oct 30 06:05:26.893154 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.893095 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4" event={"ID":"7710b253-73f0-4ee8-863a-39e6b5e472a1","Type":"ContainerDied","Data":"f0d0b6ee3135c55178d0ee87aa9dfbe5dab4e710f2e7e9c45620f9232a1048b3"} Oct 30 06:05:26.902978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.902960 5072 scope.go:122] "RemoveContainer" containerID="f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468" Oct 30 06:05:26.918617 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.918581 5072 scope.go:122] "RemoveContainer" containerID="f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468" Oct 30 06:05:26.918937 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:05:26.918891 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468\": container with ID starting with f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468 not found: ID does not exist" containerID="f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468" Oct 30 06:05:26.918937 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.918922 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468"} err="failed to get container status \"f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468\": rpc error: code = NotFound desc = could not find container \"f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468\": container with ID starting with f629ac4b0cdab89c7a35d0eeecbe62b59aa1c170b8a13e5f6596f93094516468 not found: ID does not exist" Oct 30 06:05:26.951116 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.951079 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t"] Oct 30 06:05:26.960854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.960825 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-t9c2t"] Oct 30 06:05:26.996175 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:26.996136 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4"] Oct 30 06:05:27.001451 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.001427 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-2rsh4"] Oct 30 06:05:27.561126 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.561098 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 06:05:27.647710 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.647675 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-socket\") pod \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " Oct 30 06:05:27.647890 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.647738 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-data\") pod \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " Oct 30 06:05:27.647890 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.647815 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-envoy\") pod \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " Oct 30 06:05:27.647890 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.647853 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istiod-ca-cert\") pod \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " Oct 30 06:05:27.648058 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.647992 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-credential-socket\") pod \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " Oct 30 06:05:27.648058 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.648002 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-socket" pod "e77a1aab-55dd-47d6-a80c-c5ea3df300de" (UID: "e77a1aab-55dd-47d6-a80c-c5ea3df300de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:27.648058 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.647997 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-data" pod "e77a1aab-55dd-47d6-a80c-c5ea3df300de" (UID: "e77a1aab-55dd-47d6-a80c-c5ea3df300de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:27.648308 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.648283 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-credential-socket" pod "e77a1aab-55dd-47d6-a80c-c5ea3df300de" (UID: "e77a1aab-55dd-47d6-a80c-c5ea3df300de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:27.648382 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.648311 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istiod-ca-cert" pod "e77a1aab-55dd-47d6-a80c-c5ea3df300de" (UID: "e77a1aab-55dd-47d6-a80c-c5ea3df300de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:27.648503 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.648484 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-certs\") pod \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " Oct 30 06:05:27.648581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.648560 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-podinfo\") pod \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " Oct 30 06:05:27.648701 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.648687 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-token\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-token\") pod \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " Oct 30 06:05:27.648795 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.648777 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-kube-api-access-84c2m\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-kube-api-access-84c2m\") pod \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\" (UID: \"e77a1aab-55dd-47d6-a80c-c5ea3df300de\") " Oct 30 06:05:27.648949 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.648925 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-certs" pod "e77a1aab-55dd-47d6-a80c-c5ea3df300de" (UID: "e77a1aab-55dd-47d6-a80c-c5ea3df300de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:27.649615 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.649595 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:27.649721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.649618 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:27.649721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.649632 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:27.649721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.649644 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:27.649721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.649657 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:27.651700 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.651675 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-envoy" pod "e77a1aab-55dd-47d6-a80c-c5ea3df300de" (UID: "e77a1aab-55dd-47d6-a80c-c5ea3df300de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:27.651850 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.651830 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-podinfo" pod "e77a1aab-55dd-47d6-a80c-c5ea3df300de" (UID: "e77a1aab-55dd-47d6-a80c-c5ea3df300de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:27.651912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.651893 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-token" pod "e77a1aab-55dd-47d6-a80c-c5ea3df300de" (UID: "e77a1aab-55dd-47d6-a80c-c5ea3df300de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:27.652176 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.652146 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-kube-api-access-84c2m" pod "e77a1aab-55dd-47d6-a80c-c5ea3df300de" (UID: "e77a1aab-55dd-47d6-a80c-c5ea3df300de"). InnerVolumeSpecName "kube-api-access-84c2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:27.751812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.751777 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:27.751812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.751808 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:27.751812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.751819 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84c2m\" (UniqueName: \"kubernetes.io/projected/e77a1aab-55dd-47d6-a80c-c5ea3df300de-kube-api-access-84c2m\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:27.752068 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.751828 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77a1aab-55dd-47d6-a80c-c5ea3df300de-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:27.904683 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.904652 5072 generic.go:356] "Generic (PLEG): container finished" podID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerID="a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1" exitCode=0 Oct 30 06:05:27.905200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.904715 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" Oct 30 06:05:27.905200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.904739 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" event={"ID":"e77a1aab-55dd-47d6-a80c-c5ea3df300de","Type":"ContainerDied","Data":"a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1"} Oct 30 06:05:27.905200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.904803 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g" event={"ID":"e77a1aab-55dd-47d6-a80c-c5ea3df300de","Type":"ContainerDied","Data":"6d33aeaba4bfb9c4d0049c0c5d9794053f055b730b0c7f423763de60ac815bac"} Oct 30 06:05:27.905200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.904836 5072 scope.go:122] "RemoveContainer" containerID="a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1" Oct 30 06:05:27.918619 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.918593 5072 scope.go:122] "RemoveContainer" containerID="a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1" Oct 30 06:05:27.918948 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:05:27.918927 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1\": container with ID starting with a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1 not found: ID does not exist" containerID="a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1" Oct 30 06:05:27.919017 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.918957 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1"} err="failed to get container status \"a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1\": rpc error: code = NotFound desc = could not find container \"a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1\": container with ID starting with a4c928d7be939c885f3af1d9ce680ff3068091130352095911f578665f2f96a1 not found: ID does not exist" Oct 30 06:05:27.995391 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:27.995356 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g"] Oct 30 06:05:28.007169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:28.007139 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zdt2g"] Oct 30 06:05:28.521777 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:28.521742 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" path="/var/lib/kubelet/pods/7710b253-73f0-4ee8-863a-39e6b5e472a1/volumes" Oct 30 06:05:28.522311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:28.522296 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" path="/var/lib/kubelet/pods/80e19c9f-4d40-499d-bf2e-4c92e850391b/volumes" Oct 30 06:05:28.522842 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:28.522826 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" path="/var/lib/kubelet/pods/e77a1aab-55dd-47d6-a80c-c5ea3df300de/volumes" Oct 30 06:05:30.713521 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:05:30.713478 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:05:33.137125 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:33.137087 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67"] Oct 30 06:05:33.165129 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:33.165098 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp"] Oct 30 06:05:38.153487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:38.140101 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" podUID="a13eeada-6c59-4d2f-8aa7-1935811a334b" containerName="payload-processing" containerID="cri-o://2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46" gracePeriod=25 Oct 30 06:05:38.170942 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:38.170895 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" podUID="f849af4d-72cb-4678-a0d9-168870a8d9c1" containerName="payload-pre-processing" containerID="cri-o://cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed" gracePeriod=25 Oct 30 06:05:42.192664 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.192617 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h"] Oct 30 06:05:42.193853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.193832 5072 cpu_manager.go:414] "containerMap: removing container" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" Oct 30 06:05:42.193853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.193852 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" Oct 30 06:05:42.194014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.193879 5072 cpu_manager.go:414] "containerMap: removing container" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" Oct 30 06:05:42.194014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.193886 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" Oct 30 06:05:42.194014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.193902 5072 cpu_manager.go:414] "containerMap: removing container" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" Oct 30 06:05:42.194014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.193909 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" Oct 30 06:05:42.194014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.193931 5072 cpu_manager.go:414] "containerMap: removing container" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" Oct 30 06:05:42.194014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.193938 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" Oct 30 06:05:42.194238 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.194115 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77a1aab-55dd-47d6-a80c-c5ea3df300de" containerName="istio-proxy" Oct 30 06:05:42.194238 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.194129 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e19c9f-4d40-499d-bf2e-4c92e850391b" containerName="istio-proxy" Oct 30 06:05:42.194238 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.194139 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb610158-0784-4321-abee-9f48c8ba2bba" containerName="istio-proxy" Oct 30 06:05:42.194238 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.194150 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7710b253-73f0-4ee8-863a-39e6b5e472a1" containerName="istio-proxy" Oct 30 06:05:42.198998 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.198967 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.227060 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.227029 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-6b8f87-openshift-default-dockercfg-pn4kl\"" Oct 30 06:05:42.231227 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.231198 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h"] Oct 30 06:05:42.327242 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.327192 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.327442 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.327258 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.327442 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.327402 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.327594 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.327571 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.327662 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.327603 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6d6j\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-kube-api-access-b6d6j\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.327765 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.327750 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.327843 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.327828 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.327900 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.327863 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.327900 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.327879 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.447818 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.447720 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.449048 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.449019 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.449255 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.449240 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.449375 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.449360 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.449620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.449601 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.449779 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.449762 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.449937 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.449923 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.450152 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.450136 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.450280 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.450264 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6d6j\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-kube-api-access-b6d6j\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.450544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.450438 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.450544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.449835 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.451178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.451156 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.465887 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.451754 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.466084 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.464261 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.632486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.595991 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.633654 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.633576 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.702532 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.675181 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.702935 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.685172 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6d6j\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-kube-api-access-b6d6j\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:42.831896 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:42.831608 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:43.198301 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:05:43.198273 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1a82a3_3f78_4053_a9e6_3ca484c81f3a.slice/crio-59b6aae026e7f9344070ed31157216369168a4d49e55980fd5f076237dda73a8 WatchSource:0}: Error finding container 59b6aae026e7f9344070ed31157216369168a4d49e55980fd5f076237dda73a8: Status 404 returned error can't find the container with id 59b6aae026e7f9344070ed31157216369168a4d49e55980fd5f076237dda73a8 Oct 30 06:05:43.198807 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:43.198787 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h"] Oct 30 06:05:43.201259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:43.201228 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:05:43.201355 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:43.201294 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:05:43.201355 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:43.201324 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:05:44.154695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:44.154654 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" event={"ID":"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a","Type":"ContainerStarted","Data":"4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182"} Oct 30 06:05:44.154936 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:44.154722 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" event={"ID":"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a","Type":"ContainerStarted","Data":"59b6aae026e7f9344070ed31157216369168a4d49e55980fd5f076237dda73a8"} Oct 30 06:05:44.831967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:44.831935 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:44.839021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:44.838993 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:44.886596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:44.886541 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" podStartSLOduration=2.886527349 podStartE2EDuration="2.886527349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:44.225106658 +0000 UTC m=+2103.828795916" watchObservedRunningTime="2026-10-30 06:05:44.886527349 +0000 UTC m=+2104.490216606" Oct 30 06:05:45.176954 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:45.176921 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:45.178031 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:45.177980 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:05:46.757562 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:46.757522 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" podUID="a13eeada-6c59-4d2f-8aa7-1935811a334b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:05:46.757990 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:46.757537 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" podUID="f849af4d-72cb-4678-a0d9-168870a8d9c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:05:47.829767 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:47.829723 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:05:47.869620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:47.869593 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:05:55.950589 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:55.950552 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x"] Oct 30 06:05:55.951515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:55.951445 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" podUID="711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" containerName="maas-api" containerID="cri-o://7d65256d7e012cef0179aa8df04c029727800d96015a5d1ba269ffc6f913d833" gracePeriod=30 Oct 30 06:05:56.704003 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.703928 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 06:05:56.742177 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.735381 5072 generic.go:356] "Generic (PLEG): container finished" podID="711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" containerID="7d65256d7e012cef0179aa8df04c029727800d96015a5d1ba269ffc6f913d833" exitCode=0 Oct 30 06:05:56.742177 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.735642 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" event={"ID":"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a","Type":"ContainerDied","Data":"7d65256d7e012cef0179aa8df04c029727800d96015a5d1ba269ffc6f913d833"} Oct 30 06:05:56.742177 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.735698 5072 scope.go:122] "RemoveContainer" containerID="7d65256d7e012cef0179aa8df04c029727800d96015a5d1ba269ffc6f913d833" Oct 30 06:05:56.742177 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.735863 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x" Oct 30 06:05:56.763359 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.763306 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" podUID="a13eeada-6c59-4d2f-8aa7-1935811a334b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:05:56.764370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.764336 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" podUID="f849af4d-72cb-4678-a0d9-168870a8d9c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:05:56.861605 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.861572 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-maas-api-tls\") pod \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " Oct 30 06:05:56.861810 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.861616 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-metrics-certs\") pod \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " Oct 30 06:05:56.861810 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.861664 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-openshift-service-ca\") pod \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " Oct 30 06:05:56.861810 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.861783 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-kube-api-access-g8glp\" (UniqueName: \"kubernetes.io/projected/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-kube-api-access-g8glp\") pod \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\" (UID: \"711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a\") " Oct 30 06:05:56.862113 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.862084 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-openshift-service-ca" pod "711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" (UID: "711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:56.862688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.862664 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.864248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.864224 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-metrics-certs" pod "711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" (UID: "711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:05:56.864607 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.864549 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-maas-api-tls" pod "711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" (UID: "711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:05:56.864849 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.864834 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-kube-api-access-g8glp" pod "711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" (UID: "711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a"). InnerVolumeSpecName "kube-api-access-g8glp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:56.964304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.964215 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.964304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.964250 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.964304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:56.964264 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8glp\" (UniqueName: \"kubernetes.io/projected/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a-kube-api-access-g8glp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:57.128080 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:57.128025 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x"] Oct 30 06:05:57.132847 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:57.132815 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0382c2-86559897c9-dcp5x"] Oct 30 06:05:58.527723 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.527654 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" path="/var/lib/kubelet/pods/711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a/volumes" Oct 30 06:05:58.534519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.530756 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 06:05:58.534658 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.534518 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 06:05:58.593440 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.593401 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f849af4d-72cb-4678-a0d9-168870a8d9c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f849af4d-72cb-4678-a0d9-168870a8d9c1-plugins-config-volume\") pod \"f849af4d-72cb-4678-a0d9-168870a8d9c1\" (UID: \"f849af4d-72cb-4678-a0d9-168870a8d9c1\") " Oct 30 06:05:58.593684 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.593665 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f849af4d-72cb-4678-a0d9-168870a8d9c1-kube-api-access-chp4l\" (UniqueName: \"kubernetes.io/projected/f849af4d-72cb-4678-a0d9-168870a8d9c1-kube-api-access-chp4l\") pod \"f849af4d-72cb-4678-a0d9-168870a8d9c1\" (UID: \"f849af4d-72cb-4678-a0d9-168870a8d9c1\") " Oct 30 06:05:58.593766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.593724 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13eeada-6c59-4d2f-8aa7-1935811a334b-kube-api-access-lk4th\" (UniqueName: \"kubernetes.io/projected/a13eeada-6c59-4d2f-8aa7-1935811a334b-kube-api-access-lk4th\") pod \"a13eeada-6c59-4d2f-8aa7-1935811a334b\" (UID: \"a13eeada-6c59-4d2f-8aa7-1935811a334b\") " Oct 30 06:05:58.593766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.593752 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13eeada-6c59-4d2f-8aa7-1935811a334b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13eeada-6c59-4d2f-8aa7-1935811a334b-plugins-config-volume\") pod \"a13eeada-6c59-4d2f-8aa7-1935811a334b\" (UID: \"a13eeada-6c59-4d2f-8aa7-1935811a334b\") " Oct 30 06:05:58.594967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.594050 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f849af4d-72cb-4678-a0d9-168870a8d9c1-plugins-config-volume" pod "f849af4d-72cb-4678-a0d9-168870a8d9c1" (UID: "f849af4d-72cb-4678-a0d9-168870a8d9c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:58.594967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.594314 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13eeada-6c59-4d2f-8aa7-1935811a334b-plugins-config-volume" pod "a13eeada-6c59-4d2f-8aa7-1935811a334b" (UID: "a13eeada-6c59-4d2f-8aa7-1935811a334b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:58.595425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.595406 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13eeada-6c59-4d2f-8aa7-1935811a334b-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:58.595589 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.595427 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f849af4d-72cb-4678-a0d9-168870a8d9c1-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:58.596916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.596891 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f849af4d-72cb-4678-a0d9-168870a8d9c1-kube-api-access-chp4l" pod "f849af4d-72cb-4678-a0d9-168870a8d9c1" (UID: "f849af4d-72cb-4678-a0d9-168870a8d9c1"). InnerVolumeSpecName "kube-api-access-chp4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:58.598865 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.598843 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13eeada-6c59-4d2f-8aa7-1935811a334b-kube-api-access-lk4th" pod "a13eeada-6c59-4d2f-8aa7-1935811a334b" (UID: "a13eeada-6c59-4d2f-8aa7-1935811a334b"). InnerVolumeSpecName "kube-api-access-lk4th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:58.696701 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.696668 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4th\" (UniqueName: \"kubernetes.io/projected/a13eeada-6c59-4d2f-8aa7-1935811a334b-kube-api-access-lk4th\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:58.696701 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.696698 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chp4l\" (UniqueName: \"kubernetes.io/projected/f849af4d-72cb-4678-a0d9-168870a8d9c1-kube-api-access-chp4l\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:05:58.757536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.756949 5072 generic.go:356] "Generic (PLEG): container finished" podID="a13eeada-6c59-4d2f-8aa7-1935811a334b" containerID="2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46" exitCode=0 Oct 30 06:05:58.757536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.756996 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" event={"ID":"a13eeada-6c59-4d2f-8aa7-1935811a334b","Type":"ContainerDied","Data":"2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46"} Oct 30 06:05:58.757536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.757017 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" Oct 30 06:05:58.757536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.757064 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67" event={"ID":"a13eeada-6c59-4d2f-8aa7-1935811a334b","Type":"ContainerDied","Data":"2d170baef5a749f9702c68dd7cb2c0ebaf866fd3f2eeb94a9f30b3379beb93ff"} Oct 30 06:05:58.757536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.757089 5072 scope.go:122] "RemoveContainer" containerID="2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46" Oct 30 06:05:58.759207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.759146 5072 generic.go:356] "Generic (PLEG): container finished" podID="f849af4d-72cb-4678-a0d9-168870a8d9c1" containerID="cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed" exitCode=0 Oct 30 06:05:58.759207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.759192 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" Oct 30 06:05:58.759447 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.759245 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" event={"ID":"f849af4d-72cb-4678-a0d9-168870a8d9c1","Type":"ContainerDied","Data":"cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed"} Oct 30 06:05:58.759447 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.759317 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp" event={"ID":"f849af4d-72cb-4678-a0d9-168870a8d9c1","Type":"ContainerDied","Data":"3940ac64f2d6ededf9b4fc89afb950fc0d63e596c0994caa900b22cdcffdc31e"} Oct 30 06:05:58.771792 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.771770 5072 scope.go:122] "RemoveContainer" containerID="2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46" Oct 30 06:05:58.772129 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:05:58.772105 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46\": container with ID starting with 2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46 not found: ID does not exist" containerID="2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46" Oct 30 06:05:58.772221 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.772138 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46"} err="failed to get container status \"2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46\": rpc error: code = NotFound desc = could not find container \"2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46\": container with ID starting with 2b33f896cae2dd1d2fbef12601a32f0d658af9880d9e180b81846f11a7f6ee46 not found: ID does not exist" Oct 30 06:05:58.772221 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.772158 5072 scope.go:122] "RemoveContainer" containerID="cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed" Oct 30 06:05:58.783728 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.783710 5072 scope.go:122] "RemoveContainer" containerID="cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed" Oct 30 06:05:58.784028 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:05:58.784010 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed\": container with ID starting with cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed not found: ID does not exist" containerID="cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed" Oct 30 06:05:58.784097 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.784037 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed"} err="failed to get container status \"cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed\": rpc error: code = NotFound desc = could not find container \"cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed\": container with ID starting with cf3872779c00b8da055b45cd16c4519288336e000d9a4991bcbf5ec2774918ed not found: ID does not exist" Oct 30 06:05:58.811972 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.811941 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp"] Oct 30 06:05:58.816287 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.816257 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0382c2-7c4db445ff-bmcwp"] Oct 30 06:05:58.847621 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.847583 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67"] Oct 30 06:05:58.854274 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:05:58.854240 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0382c2-6d6d5ff88d-5tm67"] Oct 30 06:06:00.576668 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:00.574090 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13eeada-6c59-4d2f-8aa7-1935811a334b" path="/var/lib/kubelet/pods/a13eeada-6c59-4d2f-8aa7-1935811a334b/volumes" Oct 30 06:06:00.576668 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:00.574879 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f849af4d-72cb-4678-a0d9-168870a8d9c1" path="/var/lib/kubelet/pods/f849af4d-72cb-4678-a0d9-168870a8d9c1/volumes" Oct 30 06:06:01.019775 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:06:01.019739 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:06:05.159080 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:05.159046 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t"] Oct 30 06:06:05.159800 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:05.159303 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" containerID="cri-o://ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302" gracePeriod=30 Oct 30 06:06:10.484533 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.484503 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 06:06:10.547533 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547499 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-envoy\") pod \"fab0bf3a-d180-4e27-a171-33b97bec9497\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " Oct 30 06:06:10.547710 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547566 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-data\") pod \"fab0bf3a-d180-4e27-a171-33b97bec9497\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " Oct 30 06:06:10.547710 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547656 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-certs\") pod \"fab0bf3a-d180-4e27-a171-33b97bec9497\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " Oct 30 06:06:10.547710 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547681 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-podinfo\") pod \"fab0bf3a-d180-4e27-a171-33b97bec9497\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " Oct 30 06:06:10.547902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547789 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-socket\") pod \"fab0bf3a-d180-4e27-a171-33b97bec9497\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " Oct 30 06:06:10.547902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547823 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-credential-socket\") pod \"fab0bf3a-d180-4e27-a171-33b97bec9497\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " Oct 30 06:06:10.547902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547866 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-kube-api-access-pzz29\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-kube-api-access-pzz29\") pod \"fab0bf3a-d180-4e27-a171-33b97bec9497\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " Oct 30 06:06:10.547902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547864 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-data" pod "fab0bf3a-d180-4e27-a171-33b97bec9497" (UID: "fab0bf3a-d180-4e27-a171-33b97bec9497"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:10.548127 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547912 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab0bf3a-d180-4e27-a171-33b97bec9497-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab0bf3a-d180-4e27-a171-33b97bec9497-istiod-ca-cert\") pod \"fab0bf3a-d180-4e27-a171-33b97bec9497\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " Oct 30 06:06:10.548127 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.547993 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-certs" pod "fab0bf3a-d180-4e27-a171-33b97bec9497" (UID: "fab0bf3a-d180-4e27-a171-33b97bec9497"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:10.548127 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.548074 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-token\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-token\") pod \"fab0bf3a-d180-4e27-a171-33b97bec9497\" (UID: \"fab0bf3a-d180-4e27-a171-33b97bec9497\") " Oct 30 06:06:10.548127 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.548100 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-credential-socket" pod "fab0bf3a-d180-4e27-a171-33b97bec9497" (UID: "fab0bf3a-d180-4e27-a171-33b97bec9497"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:10.548351 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.548263 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-socket" pod "fab0bf3a-d180-4e27-a171-33b97bec9497" (UID: "fab0bf3a-d180-4e27-a171-33b97bec9497"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:10.548414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.548364 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab0bf3a-d180-4e27-a171-33b97bec9497-istiod-ca-cert" pod "fab0bf3a-d180-4e27-a171-33b97bec9497" (UID: "fab0bf3a-d180-4e27-a171-33b97bec9497"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:10.548933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.548909 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab0bf3a-d180-4e27-a171-33b97bec9497-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.549056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.548936 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.549056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.548949 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.549056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.548962 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.549056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.548977 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.551317 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.551277 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-envoy" pod "fab0bf3a-d180-4e27-a171-33b97bec9497" (UID: "fab0bf3a-d180-4e27-a171-33b97bec9497"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:10.551403 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.551330 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-podinfo" pod "fab0bf3a-d180-4e27-a171-33b97bec9497" (UID: "fab0bf3a-d180-4e27-a171-33b97bec9497"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:06:10.551579 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.551559 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-kube-api-access-pzz29" pod "fab0bf3a-d180-4e27-a171-33b97bec9497" (UID: "fab0bf3a-d180-4e27-a171-33b97bec9497"). InnerVolumeSpecName "kube-api-access-pzz29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:10.551656 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.551602 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-token" pod "fab0bf3a-d180-4e27-a171-33b97bec9497" (UID: "fab0bf3a-d180-4e27-a171-33b97bec9497"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:10.651331 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.651296 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.651331 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.651331 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.651571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.651346 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab0bf3a-d180-4e27-a171-33b97bec9497-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.651571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:10.651360 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzz29\" (UniqueName: \"kubernetes.io/projected/fab0bf3a-d180-4e27-a171-33b97bec9497-kube-api-access-pzz29\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:06:11.060357 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.060324 5072 generic.go:356] "Generic (PLEG): container finished" podID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerID="ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302" exitCode=0 Oct 30 06:06:11.060611 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.060509 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" event={"ID":"fab0bf3a-d180-4e27-a171-33b97bec9497","Type":"ContainerDied","Data":"ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302"} Oct 30 06:06:11.060611 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.060569 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" event={"ID":"fab0bf3a-d180-4e27-a171-33b97bec9497","Type":"ContainerDied","Data":"99c93c1c792b21b2661ce3c26ac74913e012dc6ca0af3bface49b20b35a3c3ce"} Oct 30 06:06:11.060611 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.060597 5072 scope.go:122] "RemoveContainer" containerID="ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302" Oct 30 06:06:11.060802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.060767 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t" Oct 30 06:06:11.073072 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.073052 5072 scope.go:122] "RemoveContainer" containerID="e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0" Oct 30 06:06:11.083370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.083351 5072 scope.go:122] "RemoveContainer" containerID="ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302" Oct 30 06:06:11.083646 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:06:11.083625 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302\": container with ID starting with ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302 not found: ID does not exist" containerID="ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302" Oct 30 06:06:11.083722 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.083653 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302"} err="failed to get container status \"ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302\": rpc error: code = NotFound desc = could not find container \"ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302\": container with ID starting with ae8f6affcd4916e5055682cffd10bd2112c9423ac0192323ea6f0b928afa0302 not found: ID does not exist" Oct 30 06:06:11.083722 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.083670 5072 scope.go:122] "RemoveContainer" containerID="e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0" Oct 30 06:06:11.083907 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:06:11.083894 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0\": container with ID starting with e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0 not found: ID does not exist" containerID="e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0" Oct 30 06:06:11.083946 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.083912 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0"} err="failed to get container status \"e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0\": rpc error: code = NotFound desc = could not find container \"e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0\": container with ID starting with e48647caa4c6d1d13135ae8d5072e8593675e23a28b8cc64a93e6cb423f07ae0 not found: ID does not exist" Oct 30 06:06:11.126387 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.126348 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t"] Oct 30 06:06:11.131635 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:11.131608 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0382c2-openshift-default-6cddf94c6c-q7x4t"] Oct 30 06:06:12.520989 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:12.520955 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" path="/var/lib/kubelet/pods/fab0bf3a-d180-4e27-a171-33b97bec9497/volumes" Oct 30 06:06:13.670188 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.670155 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2"] Oct 30 06:06:13.671156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671084 5072 cpu_manager.go:414] "containerMap: removing container" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" Oct 30 06:06:13.671156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671105 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" Oct 30 06:06:13.671156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671135 5072 cpu_manager.go:414] "containerMap: removing container" podUID="711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" containerName="maas-api" Oct 30 06:06:13.671156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671142 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" containerName="maas-api" Oct 30 06:06:13.671156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671161 5072 cpu_manager.go:414] "containerMap: removing container" podUID="f849af4d-72cb-4678-a0d9-168870a8d9c1" containerName="payload-pre-processing" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671168 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f849af4d-72cb-4678-a0d9-168870a8d9c1" containerName="payload-pre-processing" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671195 5072 cpu_manager.go:414] "containerMap: removing container" podUID="a13eeada-6c59-4d2f-8aa7-1935811a334b" containerName="payload-processing" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671199 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13eeada-6c59-4d2f-8aa7-1935811a334b" containerName="payload-processing" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671212 5072 cpu_manager.go:414] "containerMap: removing container" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671216 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671327 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671337 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab0bf3a-d180-4e27-a171-33b97bec9497" containerName="istio-proxy" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671345 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="711d1d7d-61f0-4ad6-bfd7-64bdf2ad593a" containerName="maas-api" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671354 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f849af4d-72cb-4678-a0d9-168870a8d9c1" containerName="payload-pre-processing" Oct 30 06:06:13.671415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.671364 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13eeada-6c59-4d2f-8aa7-1935811a334b" containerName="payload-processing" Oct 30 06:06:13.677077 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.677059 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.692772 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.692750 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-080391-openshift-default-dockercfg-w5wmr\"" Oct 30 06:06:13.710509 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.710476 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2"] Oct 30 06:06:13.726346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.726292 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.726586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.726415 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12551d54-2727-4931-8ac4-4023d7613f83-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.726678 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.726583 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.726744 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.726719 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.726803 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.726749 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.726803 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.726783 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.726917 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.726805 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12551d54-2727-4931-8ac4-4023d7613f83-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.726917 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.726889 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.726917 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.726911 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5xt\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-kube-api-access-rf5xt\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.829137 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.829033 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.829319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.829204 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.829319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.829257 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.829319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.829291 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.829319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.829312 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12551d54-2727-4931-8ac4-4023d7613f83-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.829588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.829362 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.829588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.829402 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5xt\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-kube-api-access-rf5xt\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.829588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.829562 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.829746 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.829609 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12551d54-2727-4931-8ac4-4023d7613f83-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.830302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.830185 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.830302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.830219 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.830302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.830258 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.830302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.830275 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.830839 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.830817 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12551d54-2727-4931-8ac4-4023d7613f83-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.832288 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.832260 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.832557 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.832541 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12551d54-2727-4931-8ac4-4023d7613f83-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.855800 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.855763 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.856281 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.856261 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5xt\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-kube-api-access-rf5xt\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:13.992626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:13.992586 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:14.322178 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:06:14.322151 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12551d54_2727_4931_8ac4_4023d7613f83.slice/crio-7d8de0546db51aea5eb4223eb7f015d6a84b8f18456dc5fc1181c33c2e0ced38 WatchSource:0}: Error finding container 7d8de0546db51aea5eb4223eb7f015d6a84b8f18456dc5fc1181c33c2e0ced38: Status 404 returned error can't find the container with id 7d8de0546db51aea5eb4223eb7f015d6a84b8f18456dc5fc1181c33c2e0ced38 Oct 30 06:06:14.322424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:14.322404 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2"] Oct 30 06:06:14.325646 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:14.325616 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:06:14.325721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:14.325684 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:06:14.325721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:14.325712 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:06:15.105210 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:15.105170 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" event={"ID":"12551d54-2727-4931-8ac4-4023d7613f83","Type":"ContainerStarted","Data":"6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e"} Oct 30 06:06:15.105688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:15.105232 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" event={"ID":"12551d54-2727-4931-8ac4-4023d7613f83","Type":"ContainerStarted","Data":"7d8de0546db51aea5eb4223eb7f015d6a84b8f18456dc5fc1181c33c2e0ced38"} Oct 30 06:06:15.179685 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:15.179624 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" podStartSLOduration=2.179605502 podStartE2EDuration="2.179605502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:15.165394679 +0000 UTC m=+2134.769083937" watchObservedRunningTime="2026-10-30 06:06:15.179605502 +0000 UTC m=+2134.783294759" Oct 30 06:06:15.993060 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:15.993026 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:15.994505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:15.994445 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:06:15.994630 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:15.994547 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" podUID="12551d54-2727-4931-8ac4-4023d7613f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:06:16.998130 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:16.998092 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:06:16.998751 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:16.998702 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" podUID="12551d54-2727-4931-8ac4-4023d7613f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:06:17.993094 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:17.993061 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:06:17.993276 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:17.993114 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" podUID="12551d54-2727-4931-8ac4-4023d7613f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:06:18.993581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:18.993542 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:06:18.994070 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:18.993611 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" podUID="12551d54-2727-4931-8ac4-4023d7613f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:06:19.992997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:19.992964 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 06:06:19.993205 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:19.993022 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" podUID="12551d54-2727-4931-8ac4-4023d7613f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 06:06:20.998936 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:20.998904 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:20.999590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:20.999388 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:21.000072 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:21.000045 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:06:30.711848 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:06:30.711801 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:06:37.319684 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.319647 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s"] Oct 30 06:06:37.324550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.324518 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.354588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.354548 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-6b8f87-serving-cert\"" Oct 30 06:06:37.359715 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.359687 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s"] Oct 30 06:06:37.474411 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.474374 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7183d70-926a-4483-a268-3dc50b09571d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.474411 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.474410 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.474674 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.474560 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.474853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.474834 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ngw4\" (UniqueName: \"kubernetes.io/projected/b7183d70-926a-4483-a268-3dc50b09571d-kube-api-access-4ngw4\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.590017 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.589925 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.600113 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.591059 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ngw4\" (UniqueName: \"kubernetes.io/projected/b7183d70-926a-4483-a268-3dc50b09571d-kube-api-access-4ngw4\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.600113 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.599154 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7183d70-926a-4483-a268-3dc50b09571d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.600113 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.599213 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.600786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.600756 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7183d70-926a-4483-a268-3dc50b09571d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.609431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.609402 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.624481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.621109 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.627716 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.627578 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ngw4\" (UniqueName: \"kubernetes.io/projected/b7183d70-926a-4483-a268-3dc50b09571d-kube-api-access-4ngw4\") pod \"maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.641808 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.641749 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:37.968513 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:06:37.968478 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7183d70_926a_4483_a268_3dc50b09571d.slice/crio-5b6f3776f73c69219b3794e9d5a3d4b26e9bd72af9e5f73abf2b599b1dcbdd62 WatchSource:0}: Error finding container 5b6f3776f73c69219b3794e9d5a3d4b26e9bd72af9e5f73abf2b599b1dcbdd62: Status 404 returned error can't find the container with id 5b6f3776f73c69219b3794e9d5a3d4b26e9bd72af9e5f73abf2b599b1dcbdd62 Oct 30 06:06:37.968641 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:37.968537 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s"] Oct 30 06:06:38.433486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:38.430703 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" event={"ID":"b7183d70-926a-4483-a268-3dc50b09571d","Type":"ContainerStarted","Data":"5b6f3776f73c69219b3794e9d5a3d4b26e9bd72af9e5f73abf2b599b1dcbdd62"} Oct 30 06:06:39.441157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:39.441112 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" event={"ID":"b7183d70-926a-4483-a268-3dc50b09571d","Type":"ContainerStarted","Data":"6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2"} Oct 30 06:06:39.441609 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:39.441186 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:06:39.488193 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:39.488132 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" podStartSLOduration=1.947232193 podStartE2EDuration="2.48811878s" totalImagesPullingTime="540.886587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:39.485316549 +0000 UTC m=+2159.089005806" watchObservedRunningTime="2026-10-30 06:06:39.48811878 +0000 UTC m=+2159.091808036" Oct 30 06:06:45.456856 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:06:45.456823 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:07:00.720083 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:07:00.720038 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:07:03.185704 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:03.185669 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr"] Oct 30 06:07:03.186165 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:03.185920 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" podUID="37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" containerName="maas-api" containerID="cri-o://3d004fe4ad097ec9e597627771bbac46e39b8fb5b9281af1e4f04945bb14dd74" gracePeriod=30 Oct 30 06:07:04.029934 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.029843 5072 generic.go:356] "Generic (PLEG): container finished" podID="37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" containerID="3d004fe4ad097ec9e597627771bbac46e39b8fb5b9281af1e4f04945bb14dd74" exitCode=0 Oct 30 06:07:04.030289 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.030265 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" event={"ID":"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5","Type":"ContainerDied","Data":"3d004fe4ad097ec9e597627771bbac46e39b8fb5b9281af1e4f04945bb14dd74"} Oct 30 06:07:04.031019 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.030994 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" event={"ID":"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5","Type":"ContainerDied","Data":"a673244d6257c642062b6b0c91d1983a0fc648f00c909b17d4d99dc88d8a78ca"} Oct 30 06:07:04.031660 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.031634 5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a673244d6257c642062b6b0c91d1983a0fc648f00c909b17d4d99dc88d8a78ca" Oct 30 06:07:04.037579 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.035946 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:07:04.130361 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.129613 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-metrics-certs\") pod \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " Oct 30 06:07:04.130361 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.129678 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-openshift-service-ca\") pod \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " Oct 30 06:07:04.130361 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.129789 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-kube-api-access-zkr6v\" (UniqueName: \"kubernetes.io/projected/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-kube-api-access-zkr6v\") pod \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " Oct 30 06:07:04.130361 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.129866 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-maas-api-tls\") pod \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\" (UID: \"37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5\") " Oct 30 06:07:04.130361 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.130319 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-openshift-service-ca" pod "37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" (UID: "37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:04.130831 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.130751 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:04.133248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.133155 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-maas-api-tls" pod "37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" (UID: "37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:04.133248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.133202 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-metrics-certs" pod "37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" (UID: "37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:04.133248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.133213 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-kube-api-access-zkr6v" pod "37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" (UID: "37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5"). InnerVolumeSpecName "kube-api-access-zkr6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:04.231964 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.231927 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:04.231964 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.231958 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkr6v\" (UniqueName: \"kubernetes.io/projected/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-kube-api-access-zkr6v\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:04.231964 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:04.231970 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:05.038065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:05.038034 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr" Oct 30 06:07:05.101305 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:05.101271 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr"] Oct 30 06:07:05.105502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:05.105473 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-765de9-55fb975f6-qb8xr"] Oct 30 06:07:06.135553 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:06.135519 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf"] Oct 30 06:07:06.169543 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:06.169508 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7"] Oct 30 06:07:06.440767 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:06.439584 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2"] Oct 30 06:07:06.440767 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:06.439853 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" containerID="cri-o://524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b" gracePeriod=30 Oct 30 06:07:06.458115 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:06.457620 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82"] Oct 30 06:07:06.458115 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:06.457859 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" containerID="cri-o://6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320" gracePeriod=30 Oct 30 06:07:06.578527 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:06.578437 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" path="/var/lib/kubelet/pods/37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5/volumes" Oct 30 06:07:11.136300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:11.136242 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" podUID="08586f8a-ffbb-4bce-aecd-9a878faefbd6" containerName="payload-processing" containerID="cri-o://51bd16a41613a5910c655d916d8cad83fb17145084045be9fdbec99757b786fb" gracePeriod=25 Oct 30 06:07:11.169808 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:11.169766 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" podUID="492274d9-c7ca-4530-80bf-34037bb30b9c" containerName="payload-pre-processing" containerID="cri-o://b210dd9a30afe20d476fc10711cb79050ab9345f2430faffe6aaf7fa688def8d" gracePeriod=25 Oct 30 06:07:11.225585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:11.225550 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59"] Oct 30 06:07:11.225781 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:11.225744 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" containerID="cri-o://4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad" gracePeriod=30 Oct 30 06:07:11.849616 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:11.849591 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 06:07:11.902173 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:11.902144 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 06:07:12.043678 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.043591 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-podinfo\") pod \"bae92881-1919-4d77-813e-d25ccd1b31c1\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " Oct 30 06:07:12.043678 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.043665 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-envoy\") pod \"bae92881-1919-4d77-813e-d25ccd1b31c1\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " Oct 30 06:07:12.043984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.043685 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-data\") pod \"ba108819-39d4-4da9-866b-50ba6ccbfc54\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " Oct 30 06:07:12.043984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.043728 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae92881-1919-4d77-813e-d25ccd1b31c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae92881-1919-4d77-813e-d25ccd1b31c1-istiod-ca-cert\") pod \"bae92881-1919-4d77-813e-d25ccd1b31c1\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " Oct 30 06:07:12.043984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.043779 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-kube-api-access-rvwlm\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-kube-api-access-rvwlm\") pod \"ba108819-39d4-4da9-866b-50ba6ccbfc54\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " Oct 30 06:07:12.043984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.043830 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba108819-39d4-4da9-866b-50ba6ccbfc54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba108819-39d4-4da9-866b-50ba6ccbfc54-istiod-ca-cert\") pod \"ba108819-39d4-4da9-866b-50ba6ccbfc54\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " Oct 30 06:07:12.043984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.043910 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-token\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-token\") pod \"ba108819-39d4-4da9-866b-50ba6ccbfc54\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " Oct 30 06:07:12.044278 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.043993 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-socket\") pod \"ba108819-39d4-4da9-866b-50ba6ccbfc54\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " Oct 30 06:07:12.044278 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.043998 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-data" pod "ba108819-39d4-4da9-866b-50ba6ccbfc54" (UID: "ba108819-39d4-4da9-866b-50ba6ccbfc54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.044278 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044042 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-envoy\") pod \"ba108819-39d4-4da9-866b-50ba6ccbfc54\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " Oct 30 06:07:12.044278 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044085 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-certs\") pod \"ba108819-39d4-4da9-866b-50ba6ccbfc54\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " Oct 30 06:07:12.044278 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044129 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-podinfo\") pod \"ba108819-39d4-4da9-866b-50ba6ccbfc54\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " Oct 30 06:07:12.044278 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044187 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-kube-api-access-ktqkj\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-kube-api-access-ktqkj\") pod \"bae92881-1919-4d77-813e-d25ccd1b31c1\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " Oct 30 06:07:12.044278 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044247 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-credential-socket\") pod \"bae92881-1919-4d77-813e-d25ccd1b31c1\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " Oct 30 06:07:12.044687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044327 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae92881-1919-4d77-813e-d25ccd1b31c1-istiod-ca-cert" pod "bae92881-1919-4d77-813e-d25ccd1b31c1" (UID: "bae92881-1919-4d77-813e-d25ccd1b31c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:12.044687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044353 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-certs\") pod \"bae92881-1919-4d77-813e-d25ccd1b31c1\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " Oct 30 06:07:12.044687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044409 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-data\") pod \"bae92881-1919-4d77-813e-d25ccd1b31c1\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " Oct 30 06:07:12.044687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044446 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-credential-socket\") pod \"ba108819-39d4-4da9-866b-50ba6ccbfc54\" (UID: \"ba108819-39d4-4da9-866b-50ba6ccbfc54\") " Oct 30 06:07:12.044687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044566 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-socket\") pod \"bae92881-1919-4d77-813e-d25ccd1b31c1\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " Oct 30 06:07:12.044687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044632 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-token\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-token\") pod \"bae92881-1919-4d77-813e-d25ccd1b31c1\" (UID: \"bae92881-1919-4d77-813e-d25ccd1b31c1\") " Oct 30 06:07:12.045016 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044697 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba108819-39d4-4da9-866b-50ba6ccbfc54-istiod-ca-cert" pod "ba108819-39d4-4da9-866b-50ba6ccbfc54" (UID: "ba108819-39d4-4da9-866b-50ba6ccbfc54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:12.045016 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.044974 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-certs" pod "bae92881-1919-4d77-813e-d25ccd1b31c1" (UID: "bae92881-1919-4d77-813e-d25ccd1b31c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.045786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.045761 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.045786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.045787 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.046021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.045800 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae92881-1919-4d77-813e-d25ccd1b31c1-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.046021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.045814 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba108819-39d4-4da9-866b-50ba6ccbfc54-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.046179 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.046078 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-credential-socket" pod "bae92881-1919-4d77-813e-d25ccd1b31c1" (UID: "bae92881-1919-4d77-813e-d25ccd1b31c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.046260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.046202 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-certs" pod "ba108819-39d4-4da9-866b-50ba6ccbfc54" (UID: "ba108819-39d4-4da9-866b-50ba6ccbfc54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.046413 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.046341 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-credential-socket" pod "ba108819-39d4-4da9-866b-50ba6ccbfc54" (UID: "ba108819-39d4-4da9-866b-50ba6ccbfc54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.046413 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.046404 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-socket" pod "bae92881-1919-4d77-813e-d25ccd1b31c1" (UID: "bae92881-1919-4d77-813e-d25ccd1b31c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.046862 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.046695 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-socket" pod "ba108819-39d4-4da9-866b-50ba6ccbfc54" (UID: "ba108819-39d4-4da9-866b-50ba6ccbfc54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.046862 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.046728 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-data" pod "bae92881-1919-4d77-813e-d25ccd1b31c1" (UID: "bae92881-1919-4d77-813e-d25ccd1b31c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.049329 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.049307 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-token" pod "ba108819-39d4-4da9-866b-50ba6ccbfc54" (UID: "ba108819-39d4-4da9-866b-50ba6ccbfc54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:12.050475 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.050436 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-podinfo" pod "ba108819-39d4-4da9-866b-50ba6ccbfc54" (UID: "ba108819-39d4-4da9-866b-50ba6ccbfc54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:07:12.050718 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.050637 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-envoy" pod "bae92881-1919-4d77-813e-d25ccd1b31c1" (UID: "bae92881-1919-4d77-813e-d25ccd1b31c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.050718 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.050688 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-podinfo" pod "bae92881-1919-4d77-813e-d25ccd1b31c1" (UID: "bae92881-1919-4d77-813e-d25ccd1b31c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:07:12.050841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.050734 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-envoy" pod "ba108819-39d4-4da9-866b-50ba6ccbfc54" (UID: "ba108819-39d4-4da9-866b-50ba6ccbfc54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:12.050909 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.050885 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-kube-api-access-ktqkj" pod "bae92881-1919-4d77-813e-d25ccd1b31c1" (UID: "bae92881-1919-4d77-813e-d25ccd1b31c1"). InnerVolumeSpecName "kube-api-access-ktqkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:12.052503 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.052286 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-kube-api-access-rvwlm" pod "ba108819-39d4-4da9-866b-50ba6ccbfc54" (UID: "ba108819-39d4-4da9-866b-50ba6ccbfc54"). InnerVolumeSpecName "kube-api-access-rvwlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:12.052503 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.052407 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-token" pod "bae92881-1919-4d77-813e-d25ccd1b31c1" (UID: "bae92881-1919-4d77-813e-d25ccd1b31c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151216 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwlm\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-kube-api-access-rvwlm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151257 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151271 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151283 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151297 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151308 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba108819-39d4-4da9-866b-50ba6ccbfc54-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151321 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktqkj\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-kube-api-access-ktqkj\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151334 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151346 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151359 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba108819-39d4-4da9-866b-50ba6ccbfc54-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151375 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151389 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151404 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.159489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.151416 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae92881-1919-4d77-813e-d25ccd1b31c1-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.258132 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.258094 5072 generic.go:356] "Generic (PLEG): container finished" podID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerID="6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320" exitCode=0 Oct 30 06:07:12.258556 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.258532 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" event={"ID":"ba108819-39d4-4da9-866b-50ba6ccbfc54","Type":"ContainerDied","Data":"6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320"} Oct 30 06:07:12.258789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.258772 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" event={"ID":"ba108819-39d4-4da9-866b-50ba6ccbfc54","Type":"ContainerDied","Data":"e85652b63bac10bbe009114b35027726d6d8358defe218b2b9570e0650ee1a98"} Oct 30 06:07:12.260597 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.260542 5072 scope.go:122] "RemoveContainer" containerID="6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320" Oct 30 06:07:12.260947 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.260932 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82" Oct 30 06:07:12.266626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.266592 5072 generic.go:356] "Generic (PLEG): container finished" podID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerID="524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b" exitCode=0 Oct 30 06:07:12.266876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.266852 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" Oct 30 06:07:12.267072 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.266794 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" event={"ID":"bae92881-1919-4d77-813e-d25ccd1b31c1","Type":"ContainerDied","Data":"524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b"} Oct 30 06:07:12.267225 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.267208 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2" event={"ID":"bae92881-1919-4d77-813e-d25ccd1b31c1","Type":"ContainerDied","Data":"a4a822556d4a998f5d0cc9f563361f19ffdc3f8b5e87d4444c3d7352065625af"} Oct 30 06:07:12.278969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.278941 5072 scope.go:122] "RemoveContainer" containerID="6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320" Oct 30 06:07:12.279382 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:07:12.279275 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320\": container with ID starting with 6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320 not found: ID does not exist" containerID="6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320" Oct 30 06:07:12.279382 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.279316 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320"} err="failed to get container status \"6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320\": rpc error: code = NotFound desc = could not find container \"6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320\": container with ID starting with 6a74e3fcc0e6d9440067fb3e02a58fcbc12ed38b6d7b4b563070e81fdfb92320 not found: ID does not exist" Oct 30 06:07:12.279382 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.279335 5072 scope.go:122] "RemoveContainer" containerID="524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b" Oct 30 06:07:12.290530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.290509 5072 scope.go:122] "RemoveContainer" containerID="524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b" Oct 30 06:07:12.290864 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:07:12.290839 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b\": container with ID starting with 524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b not found: ID does not exist" containerID="524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b" Oct 30 06:07:12.290936 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.290871 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b"} err="failed to get container status \"524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b\": rpc error: code = NotFound desc = could not find container \"524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b\": container with ID starting with 524f85c49590555887a0d28927fda4aea9dee7544d0998f1bf53f3576328ee0b not found: ID does not exist" Oct 30 06:07:12.322926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.322846 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2"] Oct 30 06:07:12.333678 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.333647 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-98bd2"] Oct 30 06:07:12.360536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.360503 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82"] Oct 30 06:07:12.364206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.364182 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-fzq82"] Oct 30 06:07:12.522097 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.522063 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" path="/var/lib/kubelet/pods/ba108819-39d4-4da9-866b-50ba6ccbfc54/volumes" Oct 30 06:07:12.522857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:12.522838 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" path="/var/lib/kubelet/pods/bae92881-1919-4d77-813e-d25ccd1b31c1/volumes" Oct 30 06:07:13.838264 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:13.838171 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" podUID="492274d9-c7ca-4530-80bf-34037bb30b9c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:07:13.838264 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:13.838202 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" podUID="08586f8a-ffbb-4bce-aecd-9a878faefbd6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:07:16.553692 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.553665 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 06:07:16.668969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.668873 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45cfb151-b682-400e-a174-f37d2c65f08a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cfb151-b682-400e-a174-f37d2c65f08a-istio-podinfo\") pod \"45cfb151-b682-400e-a174-f37d2c65f08a\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " Oct 30 06:07:16.669162 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669025 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-socket\") pod \"45cfb151-b682-400e-a174-f37d2c65f08a\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " Oct 30 06:07:16.669162 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669123 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-credential-socket\") pod \"45cfb151-b682-400e-a174-f37d2c65f08a\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " Oct 30 06:07:16.669301 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669169 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-istio-token\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-istio-token\") pod \"45cfb151-b682-400e-a174-f37d2c65f08a\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " Oct 30 06:07:16.669301 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669266 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-certs\") pod \"45cfb151-b682-400e-a174-f37d2c65f08a\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " Oct 30 06:07:16.669414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669347 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-envoy\") pod \"45cfb151-b682-400e-a174-f37d2c65f08a\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " Oct 30 06:07:16.669493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669433 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-socket" pod "45cfb151-b682-400e-a174-f37d2c65f08a" (UID: "45cfb151-b682-400e-a174-f37d2c65f08a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.669564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669508 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45cfb151-b682-400e-a174-f37d2c65f08a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cfb151-b682-400e-a174-f37d2c65f08a-istiod-ca-cert\") pod \"45cfb151-b682-400e-a174-f37d2c65f08a\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " Oct 30 06:07:16.669631 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669574 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-kube-api-access-bbfcf\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-kube-api-access-bbfcf\") pod \"45cfb151-b682-400e-a174-f37d2c65f08a\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " Oct 30 06:07:16.669699 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669671 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-data\") pod \"45cfb151-b682-400e-a174-f37d2c65f08a\" (UID: \"45cfb151-b682-400e-a174-f37d2c65f08a\") " Oct 30 06:07:16.670196 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.669995 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-credential-socket" pod "45cfb151-b682-400e-a174-f37d2c65f08a" (UID: "45cfb151-b682-400e-a174-f37d2c65f08a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.670366 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.670110 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-certs" pod "45cfb151-b682-400e-a174-f37d2c65f08a" (UID: "45cfb151-b682-400e-a174-f37d2c65f08a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.670492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.670404 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cfb151-b682-400e-a174-f37d2c65f08a-istiod-ca-cert" pod "45cfb151-b682-400e-a174-f37d2c65f08a" (UID: "45cfb151-b682-400e-a174-f37d2c65f08a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:16.670690 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.670659 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-data" pod "45cfb151-b682-400e-a174-f37d2c65f08a" (UID: "45cfb151-b682-400e-a174-f37d2c65f08a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.671626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.671532 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45cfb151-b682-400e-a174-f37d2c65f08a-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.671626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.671559 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.671626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.671574 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.671626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.671588 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.671626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.671603 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.672019 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.671829 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45cfb151-b682-400e-a174-f37d2c65f08a-istio-podinfo" pod "45cfb151-b682-400e-a174-f37d2c65f08a" (UID: "45cfb151-b682-400e-a174-f37d2c65f08a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:07:16.672221 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.672202 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-istio-token" pod "45cfb151-b682-400e-a174-f37d2c65f08a" (UID: "45cfb151-b682-400e-a174-f37d2c65f08a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:16.672349 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.672327 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-envoy" pod "45cfb151-b682-400e-a174-f37d2c65f08a" (UID: "45cfb151-b682-400e-a174-f37d2c65f08a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.673769 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.673749 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-kube-api-access-bbfcf" pod "45cfb151-b682-400e-a174-f37d2c65f08a" (UID: "45cfb151-b682-400e-a174-f37d2c65f08a"). InnerVolumeSpecName "kube-api-access-bbfcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:16.773032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.772933 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45cfb151-b682-400e-a174-f37d2c65f08a-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.773032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.772967 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.773032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.772976 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45cfb151-b682-400e-a174-f37d2c65f08a-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.773032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:16.772986 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbfcf\" (UniqueName: \"kubernetes.io/projected/45cfb151-b682-400e-a174-f37d2c65f08a-kube-api-access-bbfcf\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:17.307752 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.307720 5072 generic.go:356] "Generic (PLEG): container finished" podID="45cfb151-b682-400e-a174-f37d2c65f08a" containerID="4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad" exitCode=0 Oct 30 06:07:17.307928 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.307769 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" Oct 30 06:07:17.307928 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.307801 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" event={"ID":"45cfb151-b682-400e-a174-f37d2c65f08a","Type":"ContainerDied","Data":"4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad"} Oct 30 06:07:17.307928 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.307860 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59" event={"ID":"45cfb151-b682-400e-a174-f37d2c65f08a","Type":"ContainerDied","Data":"4ac7b73df69248cc742ea2d7e11135c7abcb7c0c345f7bc2c93b1150e128da8a"} Oct 30 06:07:17.307928 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.307882 5072 scope.go:122] "RemoveContainer" containerID="4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad" Oct 30 06:07:17.324878 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.324855 5072 scope.go:122] "RemoveContainer" containerID="f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad" Oct 30 06:07:17.341913 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.340970 5072 scope.go:122] "RemoveContainer" containerID="4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad" Oct 30 06:07:17.341913 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:07:17.341415 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad\": container with ID starting with 4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad not found: ID does not exist" containerID="4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad" Oct 30 06:07:17.341913 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.341791 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad"} err="failed to get container status \"4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad\": rpc error: code = NotFound desc = could not find container \"4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad\": container with ID starting with 4f3aa3b3151b70546cbda27259b1cf0a7328ec78a68ec13aec5e922e73117bad not found: ID does not exist" Oct 30 06:07:17.341913 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.341815 5072 scope.go:122] "RemoveContainer" containerID="f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad" Oct 30 06:07:17.342629 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:07:17.342364 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad\": container with ID starting with f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad not found: ID does not exist" containerID="f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad" Oct 30 06:07:17.342629 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.342398 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad"} err="failed to get container status \"f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad\": rpc error: code = NotFound desc = could not find container \"f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad\": container with ID starting with f36aee6f549b2918227a7ee24804b1ca4b77da22e3a35b2a4be49982591a02ad not found: ID does not exist" Oct 30 06:07:17.382907 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.382876 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59"] Oct 30 06:07:17.395170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:17.395145 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-tgf59"] Oct 30 06:07:18.522077 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:18.522044 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" path="/var/lib/kubelet/pods/45cfb151-b682-400e-a174-f37d2c65f08a/volumes" Oct 30 06:07:23.837171 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:23.837127 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" podUID="08586f8a-ffbb-4bce-aecd-9a878faefbd6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:07:23.837651 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:23.837155 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" podUID="492274d9-c7ca-4530-80bf-34037bb30b9c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:07:28.480908 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.480874 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5"] Oct 30 06:07:28.481673 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481656 5072 cpu_manager.go:414] "containerMap: removing container" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" Oct 30 06:07:28.481673 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481673 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" Oct 30 06:07:28.481770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481695 5072 cpu_manager.go:414] "containerMap: removing container" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" Oct 30 06:07:28.481770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481700 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" Oct 30 06:07:28.481770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481717 5072 cpu_manager.go:414] "containerMap: removing container" podUID="37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" containerName="maas-api" Oct 30 06:07:28.481770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481722 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" containerName="maas-api" Oct 30 06:07:28.481770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481754 5072 cpu_manager.go:414] "containerMap: removing container" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" Oct 30 06:07:28.481770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481759 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" Oct 30 06:07:28.481967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481864 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="37dc5ac1-6e11-4bd0-b1d2-4fd52eb0baf5" containerName="maas-api" Oct 30 06:07:28.481967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481874 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" Oct 30 06:07:28.481967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481884 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae92881-1919-4d77-813e-d25ccd1b31c1" containerName="istio-proxy" Oct 30 06:07:28.481967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.481891 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba108819-39d4-4da9-866b-50ba6ccbfc54" containerName="istio-proxy" Oct 30 06:07:28.487545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.487512 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.518773 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.518745 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3abe42\"/\"default-dockercfg-z95zz\"" Oct 30 06:07:28.518945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.518832 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3abe42\"/\"kube-root-ca.crt\"" Oct 30 06:07:28.519020 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.518963 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-3abe42\"/\"sub-test-model-3abe42-kserve-self-signed-certs\"" Oct 30 06:07:28.519095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.519025 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-3abe42\"/\"openshift-service-ca.crt\"" Oct 30 06:07:28.539667 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.539630 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5"] Oct 30 06:07:28.593840 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.593809 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-model-cache\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.594008 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.593860 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-tmp-dir\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.594079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.594050 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnq6l\" (UniqueName: \"kubernetes.io/projected/b118367c-757d-4262-a1f9-341c84dc1741-kube-api-access-qnq6l\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.594264 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.594245 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b118367c-757d-4262-a1f9-341c84dc1741-tls-certs\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.594329 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.594307 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-home\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.594520 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.594485 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-dshm\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.696055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.696021 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-tmp-dir\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.696223 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.696125 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnq6l\" (UniqueName: \"kubernetes.io/projected/b118367c-757d-4262-a1f9-341c84dc1741-kube-api-access-qnq6l\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.696281 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.696253 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b118367c-757d-4262-a1f9-341c84dc1741-tls-certs\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.696324 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.696297 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-home\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.696426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.696407 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-dshm\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.696500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.696425 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-tmp-dir\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.696500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.696473 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-model-cache\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.696742 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.696715 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-home\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.696823 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.696751 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-model-cache\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.699136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.699112 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b118367c-757d-4262-a1f9-341c84dc1741-tls-certs\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.699264 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.699169 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-dshm\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.719998 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.719965 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnq6l\" (UniqueName: \"kubernetes.io/projected/b118367c-757d-4262-a1f9-341c84dc1741-kube-api-access-qnq6l\") pod \"sub-test-model-3abe42-kserve-7f55689df5-8kxt5\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:28.822528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:28.821771 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:29.260473 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:29.260423 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5"] Oct 30 06:07:29.261331 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:07:29.261303 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb118367c_757d_4262_a1f9_341c84dc1741.slice/crio-0b57adeaae3c5e803db48ddac3199c663377281e7dc084a6613b42f9be8878a5 WatchSource:0}: Error finding container 0b57adeaae3c5e803db48ddac3199c663377281e7dc084a6613b42f9be8878a5: Status 404 returned error can't find the container with id 0b57adeaae3c5e803db48ddac3199c663377281e7dc084a6613b42f9be8878a5 Oct 30 06:07:29.410033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:29.409992 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" event={"ID":"b118367c-757d-4262-a1f9-341c84dc1741","Type":"ContainerStarted","Data":"729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374"} Oct 30 06:07:29.410233 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:29.410044 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" event={"ID":"b118367c-757d-4262-a1f9-341c84dc1741","Type":"ContainerStarted","Data":"0b57adeaae3c5e803db48ddac3199c663377281e7dc084a6613b42f9be8878a5"} Oct 30 06:07:29.456507 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:29.456430 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" podStartSLOduration=1.456418011 podStartE2EDuration="1.456418011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:29.45602824 +0000 UTC m=+2209.059717497" watchObservedRunningTime="2026-10-30 06:07:29.456418011 +0000 UTC m=+2209.060107267" Oct 30 06:07:29.823089 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:29.823059 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:30.757083 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:07:30.757041 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:07:31.453482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.445770 5072 generic.go:356] "Generic (PLEG): container finished" podID="492274d9-c7ca-4530-80bf-34037bb30b9c" containerID="b210dd9a30afe20d476fc10711cb79050ab9345f2430faffe6aaf7fa688def8d" exitCode=0 Oct 30 06:07:31.453482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.445933 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" event={"ID":"492274d9-c7ca-4530-80bf-34037bb30b9c","Type":"ContainerDied","Data":"b210dd9a30afe20d476fc10711cb79050ab9345f2430faffe6aaf7fa688def8d"} Oct 30 06:07:31.453482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.448251 5072 generic.go:356] "Generic (PLEG): container finished" podID="08586f8a-ffbb-4bce-aecd-9a878faefbd6" containerID="51bd16a41613a5910c655d916d8cad83fb17145084045be9fdbec99757b786fb" exitCode=0 Oct 30 06:07:31.453482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.450590 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" event={"ID":"08586f8a-ffbb-4bce-aecd-9a878faefbd6","Type":"ContainerDied","Data":"51bd16a41613a5910c655d916d8cad83fb17145084045be9fdbec99757b786fb"} Oct 30 06:07:31.667524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.667500 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 06:07:31.671305 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.671280 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 06:07:31.776692 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.776610 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08586f8a-ffbb-4bce-aecd-9a878faefbd6-kube-api-access-xph7b\" (UniqueName: \"kubernetes.io/projected/08586f8a-ffbb-4bce-aecd-9a878faefbd6-kube-api-access-xph7b\") pod \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\" (UID: \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\") " Oct 30 06:07:31.776914 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.776743 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08586f8a-ffbb-4bce-aecd-9a878faefbd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08586f8a-ffbb-4bce-aecd-9a878faefbd6-plugins-config-volume\") pod \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\" (UID: \"08586f8a-ffbb-4bce-aecd-9a878faefbd6\") " Oct 30 06:07:31.777038 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.776994 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492274d9-c7ca-4530-80bf-34037bb30b9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492274d9-c7ca-4530-80bf-34037bb30b9c-plugins-config-volume\") pod \"492274d9-c7ca-4530-80bf-34037bb30b9c\" (UID: \"492274d9-c7ca-4530-80bf-34037bb30b9c\") " Oct 30 06:07:31.777119 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.777058 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492274d9-c7ca-4530-80bf-34037bb30b9c-kube-api-access-2blqw\" (UniqueName: \"kubernetes.io/projected/492274d9-c7ca-4530-80bf-34037bb30b9c-kube-api-access-2blqw\") pod \"492274d9-c7ca-4530-80bf-34037bb30b9c\" (UID: \"492274d9-c7ca-4530-80bf-34037bb30b9c\") " Oct 30 06:07:31.777441 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.777419 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08586f8a-ffbb-4bce-aecd-9a878faefbd6-plugins-config-volume" pod "08586f8a-ffbb-4bce-aecd-9a878faefbd6" (UID: "08586f8a-ffbb-4bce-aecd-9a878faefbd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:31.777586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.777420 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492274d9-c7ca-4530-80bf-34037bb30b9c-plugins-config-volume" pod "492274d9-c7ca-4530-80bf-34037bb30b9c" (UID: "492274d9-c7ca-4530-80bf-34037bb30b9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:31.777984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.777909 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492274d9-c7ca-4530-80bf-34037bb30b9c-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:31.777984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.777937 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08586f8a-ffbb-4bce-aecd-9a878faefbd6-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:31.779649 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.779626 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492274d9-c7ca-4530-80bf-34037bb30b9c-kube-api-access-2blqw" pod "492274d9-c7ca-4530-80bf-34037bb30b9c" (UID: "492274d9-c7ca-4530-80bf-34037bb30b9c"). InnerVolumeSpecName "kube-api-access-2blqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:31.779767 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.779726 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08586f8a-ffbb-4bce-aecd-9a878faefbd6-kube-api-access-xph7b" pod "08586f8a-ffbb-4bce-aecd-9a878faefbd6" (UID: "08586f8a-ffbb-4bce-aecd-9a878faefbd6"). InnerVolumeSpecName "kube-api-access-xph7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:31.879469 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.879427 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2blqw\" (UniqueName: \"kubernetes.io/projected/492274d9-c7ca-4530-80bf-34037bb30b9c-kube-api-access-2blqw\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:31.879723 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:31.879485 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xph7b\" (UniqueName: \"kubernetes.io/projected/08586f8a-ffbb-4bce-aecd-9a878faefbd6-kube-api-access-xph7b\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:32.460355 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.460263 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" Oct 30 06:07:32.460355 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.460293 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7" event={"ID":"492274d9-c7ca-4530-80bf-34037bb30b9c","Type":"ContainerDied","Data":"707bf26ea2984dc96e283c64703000770bc2eb3b7c590b9cc5f46ddb9b28a10c"} Oct 30 06:07:32.460933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.460366 5072 scope.go:122] "RemoveContainer" containerID="b210dd9a30afe20d476fc10711cb79050ab9345f2430faffe6aaf7fa688def8d" Oct 30 06:07:32.462683 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.462596 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" Oct 30 06:07:32.462683 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.462672 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf" event={"ID":"08586f8a-ffbb-4bce-aecd-9a878faefbd6","Type":"ContainerDied","Data":"70953831ba0666034fa4095ba70321ba7a594e26d84ce5ef502c0d15695a4dca"} Oct 30 06:07:32.479064 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.479044 5072 scope.go:122] "RemoveContainer" containerID="51bd16a41613a5910c655d916d8cad83fb17145084045be9fdbec99757b786fb" Oct 30 06:07:32.528518 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.528490 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7"] Oct 30 06:07:32.532291 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.532267 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-086c60-6f68dc97f5-vtvf7"] Oct 30 06:07:32.564755 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.564719 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf"] Oct 30 06:07:32.568997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:32.568967 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-086c60-64f68b6666-xqhlf"] Oct 30 06:07:34.549485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:34.543480 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08586f8a-ffbb-4bce-aecd-9a878faefbd6" path="/var/lib/kubelet/pods/08586f8a-ffbb-4bce-aecd-9a878faefbd6/volumes" Oct 30 06:07:34.549485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:34.544217 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492274d9-c7ca-4530-80bf-34037bb30b9c" path="/var/lib/kubelet/pods/492274d9-c7ca-4530-80bf-34037bb30b9c/volumes" Oct 30 06:07:38.844481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:38.843489 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:38.862429 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:38.862397 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:39.603565 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:39.603534 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:07:41.876195 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:41.876139 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56"] Oct 30 06:07:44.638315 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:44.638284 5072 generic.go:356] "Generic (PLEG): container finished" podID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerID="4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182" exitCode=0 Oct 30 06:07:44.638850 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:44.638372 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" event={"ID":"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a","Type":"ContainerDied","Data":"4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182"} Oct 30 06:07:44.638899 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:44.638878 5072 scope.go:122] "RemoveContainer" containerID="4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182" Oct 30 06:07:44.641040 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:44.641011 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:07:44.641148 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:44.641086 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:07:44.641148 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:44.641123 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:07:45.177612 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:45.177574 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:07:45.650976 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:45.650942 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" event={"ID":"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a","Type":"ContainerStarted","Data":"c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99"} Oct 30 06:07:45.834375 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:45.834338 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:07:45.837636 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:45.837603 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:07:45.837954 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:45.837908 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:07:46.835485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:46.834851 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:07:46.835485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:46.834933 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:07:47.835612 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:47.835567 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:07:47.848583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:47.848540 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:07:48.838877 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:48.838839 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:07:48.839573 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:48.839537 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:07:49.840334 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:49.840296 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:07:49.841170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:49.841133 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:07:50.069072 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:50.069038 5072 scope.go:122] "RemoveContainer" containerID="3d004fe4ad097ec9e597627771bbac46e39b8fb5b9281af1e4f04945bb14dd74" Oct 30 06:07:50.842957 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:50.842925 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 30 06:07:50.843518 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:50.843443 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 30 06:07:51.837815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:51.837785 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:07:51.838310 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:51.838284 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:07:51.839017 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:51.839000 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:07:52.267216 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.267177 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5"] Oct 30 06:07:52.268024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.268007 5072 cpu_manager.go:414] "containerMap: removing container" podUID="08586f8a-ffbb-4bce-aecd-9a878faefbd6" containerName="payload-processing" Oct 30 06:07:52.268079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.268026 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08586f8a-ffbb-4bce-aecd-9a878faefbd6" containerName="payload-processing" Oct 30 06:07:52.268079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.268044 5072 cpu_manager.go:414] "containerMap: removing container" podUID="492274d9-c7ca-4530-80bf-34037bb30b9c" containerName="payload-pre-processing" Oct 30 06:07:52.268079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.268049 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492274d9-c7ca-4530-80bf-34037bb30b9c" containerName="payload-pre-processing" Oct 30 06:07:52.268183 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.268089 5072 cpu_manager.go:414] "containerMap: removing container" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" Oct 30 06:07:52.268183 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.268094 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" Oct 30 06:07:52.268259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.268242 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="08586f8a-ffbb-4bce-aecd-9a878faefbd6" containerName="payload-processing" Oct 30 06:07:52.268259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.268249 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="45cfb151-b682-400e-a174-f37d2c65f08a" containerName="istio-proxy" Oct 30 06:07:52.268259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.268257 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="492274d9-c7ca-4530-80bf-34037bb30b9c" containerName="payload-pre-processing" Oct 30 06:07:52.273698 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.273682 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.314411 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.314369 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9845e266-8769-4780-9b49-46c78b07aece-istio-podinfo\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.314627 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.314555 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-istio-token\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.314627 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.314598 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9845e266-8769-4780-9b49-46c78b07aece-istiod-ca-cert\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.314732 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.314685 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.314807 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.314791 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-credential-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.314995 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.314979 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-data\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.315037 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.315026 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-certs\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.315075 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.315049 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxh6\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-kube-api-access-mbxh6\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.315243 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.315219 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-envoy\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.329096 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.329068 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5"] Oct 30 06:07:52.334063 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.334040 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d917a1-openshift-default-dockercfg-z6r76\"" Oct 30 06:07:52.416496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416444 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-data\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.416696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416517 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-certs\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.416696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416543 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxh6\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-kube-api-access-mbxh6\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.416696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416634 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-envoy\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.416696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416678 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9845e266-8769-4780-9b49-46c78b07aece-istio-podinfo\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.416967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416754 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-istio-token\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.416967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416787 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9845e266-8769-4780-9b49-46c78b07aece-istiod-ca-cert\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.416967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416830 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.416967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416884 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-data\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.416967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416947 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-certs\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.417229 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.416889 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-credential-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.417229 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.417142 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-credential-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.417476 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.417434 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.417767 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.417648 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9845e266-8769-4780-9b49-46c78b07aece-istiod-ca-cert\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.419398 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.419372 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-envoy\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.419556 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.419538 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9845e266-8769-4780-9b49-46c78b07aece-istio-podinfo\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.440204 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.440173 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-istio-token\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.440354 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.440336 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxh6\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-kube-api-access-mbxh6\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.591433 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.591344 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:52.950650 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:07:52.950614 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9845e266_8769_4780_9b49_46c78b07aece.slice/crio-35ed47b40ec56a5312e533f034e49417484afe92d394b74a92012060456eb5fd WatchSource:0}: Error finding container 35ed47b40ec56a5312e533f034e49417484afe92d394b74a92012060456eb5fd: Status 404 returned error can't find the container with id 35ed47b40ec56a5312e533f034e49417484afe92d394b74a92012060456eb5fd Oct 30 06:07:52.952113 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.952080 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5"] Oct 30 06:07:52.957391 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.957343 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:07:52.957621 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.957598 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:07:52.957761 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:52.957745 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:07:53.767689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:53.767648 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" event={"ID":"9845e266-8769-4780-9b49-46c78b07aece","Type":"ContainerStarted","Data":"425f174da38ee609a5047324bb7a4dfd7411a9e69747ae778ccf24e747f3520a"} Oct 30 06:07:53.768174 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:53.767728 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" event={"ID":"9845e266-8769-4780-9b49-46c78b07aece","Type":"ContainerStarted","Data":"35ed47b40ec56a5312e533f034e49417484afe92d394b74a92012060456eb5fd"} Oct 30 06:07:53.847850 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:53.847783 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" podStartSLOduration=1.847764181 podStartE2EDuration="1.847764181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:53.826950198 +0000 UTC m=+2233.430639478" watchObservedRunningTime="2026-10-30 06:07:53.847764181 +0000 UTC m=+2233.451453439" Oct 30 06:07:53.969790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:53.967179 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2"] Oct 30 06:07:53.969790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:53.967397 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" podUID="d66b8be3-8690-4f96-a0e1-223a035ea62e" containerName="maas-api" containerID="cri-o://147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a" gracePeriod=30 Oct 30 06:07:54.716695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.716212 5072 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:07:54.716695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.716289 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:07:54.729118 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.718956 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:07:54.793536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.793165 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 06:07:54.841481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.837434 5072 generic.go:356] "Generic (PLEG): container finished" podID="d66b8be3-8690-4f96-a0e1-223a035ea62e" containerID="147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a" exitCode=0 Oct 30 06:07:54.841481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.838599 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" Oct 30 06:07:54.841481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.838761 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" event={"ID":"d66b8be3-8690-4f96-a0e1-223a035ea62e","Type":"ContainerDied","Data":"147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a"} Oct 30 06:07:54.841481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.838804 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2" event={"ID":"d66b8be3-8690-4f96-a0e1-223a035ea62e","Type":"ContainerDied","Data":"4a2847215c829029ae7c67b51afbcc2c4f4b5c0aa605ce8622c53c6b67de1967"} Oct 30 06:07:54.841481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.838832 5072 scope.go:122] "RemoveContainer" containerID="147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a" Oct 30 06:07:54.907303 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.907270 5072 scope.go:122] "RemoveContainer" containerID="147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a" Oct 30 06:07:54.907827 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:07:54.907802 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a\": container with ID starting with 147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a not found: ID does not exist" containerID="147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a" Oct 30 06:07:54.907936 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.907838 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a"} err="failed to get container status \"147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a\": rpc error: code = NotFound desc = could not find container \"147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a\": container with ID starting with 147228bd8024099999b6c74bd9e9f9938071b9ce8e9329d2d297459e08b12a8a not found: ID does not exist" Oct 30 06:07:54.948484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.932018 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-metrics-certs\") pod \"d66b8be3-8690-4f96-a0e1-223a035ea62e\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " Oct 30 06:07:54.948484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.932094 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-maas-api-tls\") pod \"d66b8be3-8690-4f96-a0e1-223a035ea62e\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " Oct 30 06:07:54.948484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.932177 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d66b8be3-8690-4f96-a0e1-223a035ea62e-kube-api-access-qfznq\" (UniqueName: \"kubernetes.io/projected/d66b8be3-8690-4f96-a0e1-223a035ea62e-kube-api-access-qfznq\") pod \"d66b8be3-8690-4f96-a0e1-223a035ea62e\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " Oct 30 06:07:54.948484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.932395 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d66b8be3-8690-4f96-a0e1-223a035ea62e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b8be3-8690-4f96-a0e1-223a035ea62e-openshift-service-ca\") pod \"d66b8be3-8690-4f96-a0e1-223a035ea62e\" (UID: \"d66b8be3-8690-4f96-a0e1-223a035ea62e\") " Oct 30 06:07:54.957475 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.957285 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66b8be3-8690-4f96-a0e1-223a035ea62e-openshift-service-ca" pod "d66b8be3-8690-4f96-a0e1-223a035ea62e" (UID: "d66b8be3-8690-4f96-a0e1-223a035ea62e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:54.961553 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.958246 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66b8be3-8690-4f96-a0e1-223a035ea62e-kube-api-access-qfznq" pod "d66b8be3-8690-4f96-a0e1-223a035ea62e" (UID: "d66b8be3-8690-4f96-a0e1-223a035ea62e"). InnerVolumeSpecName "kube-api-access-qfznq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:54.962902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.962863 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-maas-api-tls" pod "d66b8be3-8690-4f96-a0e1-223a035ea62e" (UID: "d66b8be3-8690-4f96-a0e1-223a035ea62e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:54.969973 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:54.969897 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-metrics-certs" pod "d66b8be3-8690-4f96-a0e1-223a035ea62e" (UID: "d66b8be3-8690-4f96-a0e1-223a035ea62e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:55.045417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:55.045342 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66b8be3-8690-4f96-a0e1-223a035ea62e-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:55.045417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:55.045379 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:55.045417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:55.045394 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d66b8be3-8690-4f96-a0e1-223a035ea62e-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:55.045417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:55.045406 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfznq\" (UniqueName: \"kubernetes.io/projected/d66b8be3-8690-4f96-a0e1-223a035ea62e-kube-api-access-qfznq\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:55.253314 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:55.253228 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2"] Oct 30 06:07:55.264433 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:55.264402 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-086c60-58f694c6d4-8qzx2"] Oct 30 06:07:55.593565 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:55.593483 5072 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:07:55.593565 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:55.593548 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:07:56.547938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:56.547899 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d66b8be3-8690-4f96-a0e1-223a035ea62e" path="/var/lib/kubelet/pods/d66b8be3-8690-4f96-a0e1-223a035ea62e/volumes" Oct 30 06:07:56.593980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:56.593946 5072 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:07:56.594158 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:56.594008 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:07:56.942803 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:56.942665 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" podUID="7ee09dc1-83a8-40af-9a26-7ac225f8506d" containerName="main" containerID="cri-o://35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204" gracePeriod=45 Oct 30 06:07:57.066942 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.066911 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" podUID="7ee09dc1-83a8-40af-9a26-7ac225f8506d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.171:8000/ready\": dial tcp 10.132.0.171:8000: connect: connection refused" Oct 30 06:07:57.298889 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.298864 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:07:57.430483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.423968 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee09dc1-83a8-40af-9a26-7ac225f8506d-kube-api-access-xsb7p\" (UniqueName: \"kubernetes.io/projected/7ee09dc1-83a8-40af-9a26-7ac225f8506d-kube-api-access-xsb7p\") pod \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " Oct 30 06:07:57.430483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.424077 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tls-certs\") pod \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " Oct 30 06:07:57.430483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.424151 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-home\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-home\") pod \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " Oct 30 06:07:57.430483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.424288 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tmp-dir\") pod \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " Oct 30 06:07:57.430483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.424424 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-dshm\") pod \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " Oct 30 06:07:57.430483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.424492 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-model-cache\") pod \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\" (UID: \"7ee09dc1-83a8-40af-9a26-7ac225f8506d\") " Oct 30 06:07:57.431245 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.431214 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tmp-dir" pod "7ee09dc1-83a8-40af-9a26-7ac225f8506d" (UID: "7ee09dc1-83a8-40af-9a26-7ac225f8506d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:57.441494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.437542 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-model-cache" pod "7ee09dc1-83a8-40af-9a26-7ac225f8506d" (UID: "7ee09dc1-83a8-40af-9a26-7ac225f8506d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:57.441494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.437852 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee09dc1-83a8-40af-9a26-7ac225f8506d-kube-api-access-xsb7p" pod "7ee09dc1-83a8-40af-9a26-7ac225f8506d" (UID: "7ee09dc1-83a8-40af-9a26-7ac225f8506d"). InnerVolumeSpecName "kube-api-access-xsb7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:57.441494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.438063 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-home" pod "7ee09dc1-83a8-40af-9a26-7ac225f8506d" (UID: "7ee09dc1-83a8-40af-9a26-7ac225f8506d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:57.445413 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.445385 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-dshm" pod "7ee09dc1-83a8-40af-9a26-7ac225f8506d" (UID: "7ee09dc1-83a8-40af-9a26-7ac225f8506d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:57.445607 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.445532 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tls-certs" pod "7ee09dc1-83a8-40af-9a26-7ac225f8506d" (UID: "7ee09dc1-83a8-40af-9a26-7ac225f8506d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:57.543558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.536655 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:57.543558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.536691 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:57.543558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.536709 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsb7p\" (UniqueName: \"kubernetes.io/projected/7ee09dc1-83a8-40af-9a26-7ac225f8506d-kube-api-access-xsb7p\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:57.543558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.536723 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:57.543558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.536737 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:57.543558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.536750 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee09dc1-83a8-40af-9a26-7ac225f8506d-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:07:57.594882 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.593631 5072 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:07:57.594882 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.593692 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:07:57.874266 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.874232 5072 generic.go:356] "Generic (PLEG): container finished" podID="7ee09dc1-83a8-40af-9a26-7ac225f8506d" containerID="35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204" exitCode=0 Oct 30 06:07:57.874476 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.874300 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" event={"ID":"7ee09dc1-83a8-40af-9a26-7ac225f8506d","Type":"ContainerDied","Data":"35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204"} Oct 30 06:07:57.874476 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.874351 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" event={"ID":"7ee09dc1-83a8-40af-9a26-7ac225f8506d","Type":"ContainerDied","Data":"a31ae4d1f6229f7f76cc1f4472ccc381f47e03cf8f76f47365ef085a0044ca1d"} Oct 30 06:07:57.874476 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.874374 5072 scope.go:122] "RemoveContainer" containerID="35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204" Oct 30 06:07:57.877627 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.875562 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56" Oct 30 06:07:57.887780 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.887761 5072 scope.go:122] "RemoveContainer" containerID="35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204" Oct 30 06:07:57.888225 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:07:57.888205 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204\": container with ID starting with 35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204 not found: ID does not exist" containerID="35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204" Oct 30 06:07:57.888347 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.888328 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204"} err="failed to get container status \"35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204\": rpc error: code = NotFound desc = could not find container \"35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204\": container with ID starting with 35adbf312c5247c83e4c7818d77c2a76307f8b6e685a7fbfca642305cc718204 not found: ID does not exist" Oct 30 06:07:57.945620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.943372 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56"] Oct 30 06:07:57.956385 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:57.956334 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/auth-test-model-02ceed-kserve-686fccf5fc-f9h56"] Oct 30 06:07:58.566351 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:58.566312 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee09dc1-83a8-40af-9a26-7ac225f8506d" path="/var/lib/kubelet/pods/7ee09dc1-83a8-40af-9a26-7ac225f8506d/volumes" Oct 30 06:07:58.593333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:58.593304 5072 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:07:58.593553 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:58.593354 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:07:59.594087 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:59.594053 5072 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:07:59.594640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:07:59.594110 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 06:08:00.598993 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:00.598957 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:08:00.599508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:00.599488 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:08:00.600246 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:00.600225 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:08:00.767548 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:00.767448 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:08:02.103497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:02.103443 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5"] Oct 30 06:08:02.104106 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:02.103778 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" containerID="cri-o://cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d" gracePeriod=30 Oct 30 06:08:02.112651 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:02.112617 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5"] Oct 30 06:08:02.112887 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:02.112860 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerName="istio-proxy" containerID="cri-o://3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4" gracePeriod=30 Oct 30 06:08:02.120977 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:02.120936 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx"] Oct 30 06:08:02.121494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:02.121413 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" containerID="cri-o://a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14" gracePeriod=30 Oct 30 06:08:07.459723 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.459691 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:08:07.489628 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.489596 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 06:08:07.497382 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.497350 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-envoy\") pod \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " Oct 30 06:08:07.497580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.497451 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-credential-socket\") pod \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " Oct 30 06:08:07.497813 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.497784 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-credential-socket" pod "715162c8-0a95-4c0d-b15e-3e57c3b707f3" (UID: "715162c8-0a95-4c0d-b15e-3e57c3b707f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.497914 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.497879 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-kube-api-access-5jdnm\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-kube-api-access-5jdnm\") pod \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " Oct 30 06:08:07.497969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.497947 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-certs\") pod \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " Oct 30 06:08:07.498039 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.498022 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-socket\") pod \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " Oct 30 06:08:07.498126 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.498110 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istiod-ca-cert\") pod \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " Oct 30 06:08:07.499159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.498206 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-podinfo\") pod \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " Oct 30 06:08:07.499159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.498250 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-token\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-token\") pod \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " Oct 30 06:08:07.499159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.498345 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-data\") pod \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\" (UID: \"715162c8-0a95-4c0d-b15e-3e57c3b707f3\") " Oct 30 06:08:07.499159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.498350 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-certs" pod "715162c8-0a95-4c0d-b15e-3e57c3b707f3" (UID: "715162c8-0a95-4c0d-b15e-3e57c3b707f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.499159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.498660 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-data" pod "715162c8-0a95-4c0d-b15e-3e57c3b707f3" (UID: "715162c8-0a95-4c0d-b15e-3e57c3b707f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.499159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.498672 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istiod-ca-cert" pod "715162c8-0a95-4c0d-b15e-3e57c3b707f3" (UID: "715162c8-0a95-4c0d-b15e-3e57c3b707f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:07.499159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.499126 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-socket" pod "715162c8-0a95-4c0d-b15e-3e57c3b707f3" (UID: "715162c8-0a95-4c0d-b15e-3e57c3b707f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.499740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.499163 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.499740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.499303 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.499740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.499318 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.499740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.499332 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.503803 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.503779 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-podinfo" pod "715162c8-0a95-4c0d-b15e-3e57c3b707f3" (UID: "715162c8-0a95-4c0d-b15e-3e57c3b707f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:07.504058 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.503952 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-token" pod "715162c8-0a95-4c0d-b15e-3e57c3b707f3" (UID: "715162c8-0a95-4c0d-b15e-3e57c3b707f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.504312 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.504272 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-kube-api-access-5jdnm" pod "715162c8-0a95-4c0d-b15e-3e57c3b707f3" (UID: "715162c8-0a95-4c0d-b15e-3e57c3b707f3"). InnerVolumeSpecName "kube-api-access-5jdnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.505442 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.505422 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-envoy" pod "715162c8-0a95-4c0d-b15e-3e57c3b707f3" (UID: "715162c8-0a95-4c0d-b15e-3e57c3b707f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.550823 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.550794 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:08:07.601001 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.600961 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-envoy\") pod \"d7f17885-3892-490b-bc8b-8499db744394\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " Oct 30 06:08:07.601191 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601038 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-socket\") pod \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " Oct 30 06:08:07.601191 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601071 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-envoy\") pod \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " Oct 30 06:08:07.601191 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601133 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-socket\") pod \"d7f17885-3892-490b-bc8b-8499db744394\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " Oct 30 06:08:07.601363 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601319 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-certs\") pod \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " Oct 30 06:08:07.601418 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601370 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-kube-api-access-whts4\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-kube-api-access-whts4\") pod \"d7f17885-3892-490b-bc8b-8499db744394\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " Oct 30 06:08:07.601418 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601407 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-istio-token\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-istio-token\") pod \"d7f17885-3892-490b-bc8b-8499db744394\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " Oct 30 06:08:07.601619 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601551 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-podinfo\") pod \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " Oct 30 06:08:07.601619 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601587 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f17885-3892-490b-bc8b-8499db744394-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f17885-3892-490b-bc8b-8499db744394-istiod-ca-cert\") pod \"d7f17885-3892-490b-bc8b-8499db744394\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " Oct 30 06:08:07.601619 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601615 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-credential-socket\") pod \"d7f17885-3892-490b-bc8b-8499db744394\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " Oct 30 06:08:07.601811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601677 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-kube-api-access-jpfsh\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-kube-api-access-jpfsh\") pod \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " Oct 30 06:08:07.601811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601726 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istiod-ca-cert\") pod \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " Oct 30 06:08:07.601811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601776 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-credential-socket\") pod \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " Oct 30 06:08:07.601966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601826 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-token\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-token\") pod \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " Oct 30 06:08:07.601966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601877 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-certs\") pod \"d7f17885-3892-490b-bc8b-8499db744394\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " Oct 30 06:08:07.601966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601955 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-data\") pod \"d7f17885-3892-490b-bc8b-8499db744394\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " Oct 30 06:08:07.602118 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.601981 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7f17885-3892-490b-bc8b-8499db744394-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f17885-3892-490b-bc8b-8499db744394-istio-podinfo\") pod \"d7f17885-3892-490b-bc8b-8499db744394\" (UID: \"d7f17885-3892-490b-bc8b-8499db744394\") " Oct 30 06:08:07.602118 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.602039 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-data\") pod \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\" (UID: \"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72\") " Oct 30 06:08:07.603759 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.603197 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-certs" pod "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" (UID: "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.603759 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.603234 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istiod-ca-cert" pod "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" (UID: "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:07.603759 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.603491 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-credential-socket" pod "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" (UID: "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.606499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.604588 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-socket" pod "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" (UID: "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.606933 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-socket" pod "d7f17885-3892-490b-bc8b-8499db744394" (UID: "d7f17885-3892-490b-bc8b-8499db744394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607199 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607221 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607235 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607247 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607259 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607272 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607284 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdnm\" (UniqueName: \"kubernetes.io/projected/715162c8-0a95-4c0d-b15e-3e57c3b707f3-kube-api-access-5jdnm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607297 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607308 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715162c8-0a95-4c0d-b15e-3e57c3b707f3-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.607320 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.608328 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-kube-api-access-whts4" pod "d7f17885-3892-490b-bc8b-8499db744394" (UID: "d7f17885-3892-490b-bc8b-8499db744394"). InnerVolumeSpecName "kube-api-access-whts4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.608884 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f17885-3892-490b-bc8b-8499db744394-istiod-ca-cert" pod "d7f17885-3892-490b-bc8b-8499db744394" (UID: "d7f17885-3892-490b-bc8b-8499db744394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.609571 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-data" pod "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" (UID: "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.609770 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-certs" pod "d7f17885-3892-490b-bc8b-8499db744394" (UID: "d7f17885-3892-490b-bc8b-8499db744394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.609878 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-data" pod "d7f17885-3892-490b-bc8b-8499db744394" (UID: "d7f17885-3892-490b-bc8b-8499db744394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.609993 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-credential-socket" pod "d7f17885-3892-490b-bc8b-8499db744394" (UID: "d7f17885-3892-490b-bc8b-8499db744394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.613536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.611886 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7f17885-3892-490b-bc8b-8499db744394-istio-podinfo" pod "d7f17885-3892-490b-bc8b-8499db744394" (UID: "d7f17885-3892-490b-bc8b-8499db744394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:07.628874 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.628818 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-envoy" pod "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" (UID: "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.628874 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.628830 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-envoy" pod "d7f17885-3892-490b-bc8b-8499db744394" (UID: "d7f17885-3892-490b-bc8b-8499db744394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.628874 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.628848 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-istio-token" pod "d7f17885-3892-490b-bc8b-8499db744394" (UID: "d7f17885-3892-490b-bc8b-8499db744394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.628874 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.628832 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-kube-api-access-jpfsh" pod "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" (UID: "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72"). InnerVolumeSpecName "kube-api-access-jpfsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.629223 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.628945 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-token" pod "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" (UID: "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.630398 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.630373 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-podinfo" pod "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" (UID: "6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:07.708835 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708751 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.708835 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708787 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.708835 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708800 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.708835 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708812 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f17885-3892-490b-bc8b-8499db744394-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.708835 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708825 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.708835 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708836 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.709260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708846 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.709260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708859 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whts4\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-kube-api-access-whts4\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.709260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708870 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f17885-3892-490b-bc8b-8499db744394-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.709260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708885 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.709260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708895 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f17885-3892-490b-bc8b-8499db744394-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.709260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708905 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f17885-3892-490b-bc8b-8499db744394-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.709260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.708913 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpfsh\" (UniqueName: \"kubernetes.io/projected/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72-kube-api-access-jpfsh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.987915 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.987820 5072 generic.go:356] "Generic (PLEG): container finished" podID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerID="3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4" exitCode=0 Oct 30 06:08:07.987915 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.987884 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" Oct 30 06:08:07.988220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.987911 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" event={"ID":"715162c8-0a95-4c0d-b15e-3e57c3b707f3","Type":"ContainerDied","Data":"3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4"} Oct 30 06:08:07.988220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.987970 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5" event={"ID":"715162c8-0a95-4c0d-b15e-3e57c3b707f3","Type":"ContainerDied","Data":"a2bc7afdc85a97428028c2548b534d7d6d2c9f043dc74309f001e58480e16e56"} Oct 30 06:08:07.988220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.987992 5072 scope.go:122] "RemoveContainer" containerID="3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4" Oct 30 06:08:07.989951 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.989919 5072 generic.go:356] "Generic (PLEG): container finished" podID="d7f17885-3892-490b-bc8b-8499db744394" containerID="a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14" exitCode=0 Oct 30 06:08:07.990079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.989980 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" Oct 30 06:08:07.990547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.990171 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" event={"ID":"d7f17885-3892-490b-bc8b-8499db744394","Type":"ContainerDied","Data":"a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14"} Oct 30 06:08:07.990672 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.990588 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx" event={"ID":"d7f17885-3892-490b-bc8b-8499db744394","Type":"ContainerDied","Data":"d6604e6bccc9d2821a7c4c429d5170ad8e1ab52a0f1d2934c467ffae6e3be9e1"} Oct 30 06:08:08.003707 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.994020 5072 generic.go:356] "Generic (PLEG): container finished" podID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerID="cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d" exitCode=0 Oct 30 06:08:08.003707 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.994067 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" event={"ID":"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72","Type":"ContainerDied","Data":"cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d"} Oct 30 06:08:08.003707 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.994112 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" event={"ID":"6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72","Type":"ContainerDied","Data":"2469853d3f35c258e019a056fef6db2e3f013769e1340342e8be8a84ca63824f"} Oct 30 06:08:08.003707 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:07.994200 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5" Oct 30 06:08:08.032303 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.028585 5072 scope.go:122] "RemoveContainer" containerID="3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4" Oct 30 06:08:08.046939 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:08.046812 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4\": container with ID starting with 3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4 not found: ID does not exist" containerID="3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4" Oct 30 06:08:08.046939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.046857 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4"} err="failed to get container status \"3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4\": rpc error: code = NotFound desc = could not find container \"3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4\": container with ID starting with 3a5ecc36d8f89a6d1f8ed80db64ce7ee0572fd636248142a6a988274704e5ec4 not found: ID does not exist" Oct 30 06:08:08.046939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.046881 5072 scope.go:122] "RemoveContainer" containerID="a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14" Oct 30 06:08:08.126479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.120169 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5"] Oct 30 06:08:08.153713 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.149196 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-7mqf5"] Oct 30 06:08:08.265069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.260278 5072 scope.go:122] "RemoveContainer" containerID="a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14" Oct 30 06:08:08.265069 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:08.260978 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14\": container with ID starting with a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14 not found: ID does not exist" containerID="a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14" Oct 30 06:08:08.265069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.261016 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14"} err="failed to get container status \"a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14\": rpc error: code = NotFound desc = could not find container \"a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14\": container with ID starting with a8689143310eea49a4800582fcc83bf1441239486a00c43abb6d8bbfa5d7ec14 not found: ID does not exist" Oct 30 06:08:08.265069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.261041 5072 scope.go:122] "RemoveContainer" containerID="cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d" Oct 30 06:08:08.271587 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.271527 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5"] Oct 30 06:08:08.279479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.275768 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-8p2w5"] Oct 30 06:08:08.292479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.284513 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx"] Oct 30 06:08:08.378443 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.378361 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-ct5gx"] Oct 30 06:08:08.423364 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.423330 5072 scope.go:122] "RemoveContainer" containerID="6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48" Oct 30 06:08:08.444663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.443673 5072 scope.go:122] "RemoveContainer" containerID="cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d" Oct 30 06:08:08.444663 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:08.444213 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d\": container with ID starting with cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d not found: ID does not exist" containerID="cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d" Oct 30 06:08:08.444663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.444251 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d"} err="failed to get container status \"cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d\": rpc error: code = NotFound desc = could not find container \"cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d\": container with ID starting with cd888841bce70f98e183ca6469baa1fdd38e1191536574f930fbc349c10c786d not found: ID does not exist" Oct 30 06:08:08.444663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.444281 5072 scope.go:122] "RemoveContainer" containerID="6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48" Oct 30 06:08:08.444663 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:08.444623 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48\": container with ID starting with 6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48 not found: ID does not exist" containerID="6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48" Oct 30 06:08:08.444663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.444651 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48"} err="failed to get container status \"6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48\": rpc error: code = NotFound desc = could not find container \"6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48\": container with ID starting with 6bb93e2e25919be0c2cafbb5cb31249f907df13658e2ead40eecc8eb0a2bdd48 not found: ID does not exist" Oct 30 06:08:08.525710 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.525620 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" path="/var/lib/kubelet/pods/6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72/volumes" Oct 30 06:08:08.526403 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.526380 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" path="/var/lib/kubelet/pods/715162c8-0a95-4c0d-b15e-3e57c3b707f3/volumes" Oct 30 06:08:08.527069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:08.527047 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f17885-3892-490b-bc8b-8499db744394" path="/var/lib/kubelet/pods/d7f17885-3892-490b-bc8b-8499db744394/volumes" Oct 30 06:08:09.788945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:09.788899 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj"] Oct 30 06:08:24.819329 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:24.819281 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" podUID="283adf34-7bce-4a42-9a6a-fbd8009ca69a" containerName="main" containerID="cri-o://e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7" gracePeriod=45 Oct 30 06:08:25.132573 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.132541 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:08:25.261270 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.260838 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tls-certs\" (UniqueName: \"kubernetes.io/secret/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tls-certs\") pod \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " Oct 30 06:08:25.261570 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.261546 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-dshm\") pod \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " Oct 30 06:08:25.261822 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.261802 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tmp-dir\") pod \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " Oct 30 06:08:25.261995 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.261978 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-home\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-home\") pod \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " Oct 30 06:08:25.262295 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.262274 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283adf34-7bce-4a42-9a6a-fbd8009ca69a-kube-api-access-mqzq5\" (UniqueName: \"kubernetes.io/projected/283adf34-7bce-4a42-9a6a-fbd8009ca69a-kube-api-access-mqzq5\") pod \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " Oct 30 06:08:25.263733 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.263710 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-model-cache\") pod \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\" (UID: \"283adf34-7bce-4a42-9a6a-fbd8009ca69a\") " Oct 30 06:08:25.264901 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.263592 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tmp-dir" pod "283adf34-7bce-4a42-9a6a-fbd8009ca69a" (UID: "283adf34-7bce-4a42-9a6a-fbd8009ca69a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:25.266038 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.266015 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-model-cache" pod "283adf34-7bce-4a42-9a6a-fbd8009ca69a" (UID: "283adf34-7bce-4a42-9a6a-fbd8009ca69a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:25.278766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.276935 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-home" pod "283adf34-7bce-4a42-9a6a-fbd8009ca69a" (UID: "283adf34-7bce-4a42-9a6a-fbd8009ca69a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:25.278766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.278200 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-dshm" pod "283adf34-7bce-4a42-9a6a-fbd8009ca69a" (UID: "283adf34-7bce-4a42-9a6a-fbd8009ca69a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:25.282825 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.282798 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tls-certs" pod "283adf34-7bce-4a42-9a6a-fbd8009ca69a" (UID: "283adf34-7bce-4a42-9a6a-fbd8009ca69a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:25.283853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.283819 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283adf34-7bce-4a42-9a6a-fbd8009ca69a-kube-api-access-mqzq5" pod "283adf34-7bce-4a42-9a6a-fbd8009ca69a" (UID: "283adf34-7bce-4a42-9a6a-fbd8009ca69a"). InnerVolumeSpecName "kube-api-access-mqzq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:25.308713 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.308679 5072 generic.go:356] "Generic (PLEG): container finished" podID="283adf34-7bce-4a42-9a6a-fbd8009ca69a" containerID="e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7" exitCode=0 Oct 30 06:08:25.308895 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.308731 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" Oct 30 06:08:25.308949 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.308903 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" event={"ID":"283adf34-7bce-4a42-9a6a-fbd8009ca69a","Type":"ContainerDied","Data":"e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7"} Oct 30 06:08:25.308949 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.308936 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj" event={"ID":"283adf34-7bce-4a42-9a6a-fbd8009ca69a","Type":"ContainerDied","Data":"c5798ed2459daafa420976a382cbe5c99f442ec761633ac24a8ed57b5e149f96"} Oct 30 06:08:25.309025 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.308954 5072 scope.go:122] "RemoveContainer" containerID="e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7" Oct 30 06:08:25.322218 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.322196 5072 scope.go:122] "RemoveContainer" containerID="e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7" Oct 30 06:08:25.322571 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:25.322543 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7\": container with ID starting with e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7 not found: ID does not exist" containerID="e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7" Oct 30 06:08:25.322657 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.322585 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7"} err="failed to get container status \"e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7\": rpc error: code = NotFound desc = could not find container \"e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7\": container with ID starting with e9fa6e961dbf86d0e0851d78d6d364261b60ca86ba42bda5f2d58cac6071c7e7 not found: ID does not exist" Oct 30 06:08:25.363505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.363474 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj"] Oct 30 06:08:25.365766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.365736 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:25.365766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.365767 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:25.365938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.365782 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqzq5\" (UniqueName: \"kubernetes.io/projected/283adf34-7bce-4a42-9a6a-fbd8009ca69a-kube-api-access-mqzq5\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:25.365938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.365795 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:25.365938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.365807 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/283adf34-7bce-4a42-9a6a-fbd8009ca69a-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:25.365938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.365818 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/283adf34-7bce-4a42-9a6a-fbd8009ca69a-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:25.370226 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:25.370203 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-482468/test-model-482468-kserve-6959c6c555-zfxnj"] Oct 30 06:08:26.525871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:26.525832 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283adf34-7bce-4a42-9a6a-fbd8009ca69a" path="/var/lib/kubelet/pods/283adf34-7bce-4a42-9a6a-fbd8009ca69a/volumes" Oct 30 06:08:29.025550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:29.025512 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb"] Oct 30 06:08:30.714484 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:30.714431 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:08:33.714640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:33.714591 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4"] Oct 30 06:08:33.715170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:33.714840 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" podUID="d706291d-60bd-427e-81b2-98523f2f5d2f" containerName="maas-api" containerID="cri-o://d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70" gracePeriod=30 Oct 30 06:08:34.174399 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.173886 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:08:34.370484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.365356 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-metrics-certs\") pod \"d706291d-60bd-427e-81b2-98523f2f5d2f\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " Oct 30 06:08:34.370484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.365495 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-maas-api-tls\") pod \"d706291d-60bd-427e-81b2-98523f2f5d2f\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " Oct 30 06:08:34.402644 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.393298 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d706291d-60bd-427e-81b2-98523f2f5d2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d706291d-60bd-427e-81b2-98523f2f5d2f-openshift-service-ca\") pod \"d706291d-60bd-427e-81b2-98523f2f5d2f\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " Oct 30 06:08:34.402644 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.393440 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d706291d-60bd-427e-81b2-98523f2f5d2f-kube-api-access-5crbp\" (UniqueName: \"kubernetes.io/projected/d706291d-60bd-427e-81b2-98523f2f5d2f-kube-api-access-5crbp\") pod \"d706291d-60bd-427e-81b2-98523f2f5d2f\" (UID: \"d706291d-60bd-427e-81b2-98523f2f5d2f\") " Oct 30 06:08:34.402644 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.399353 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d706291d-60bd-427e-81b2-98523f2f5d2f-openshift-service-ca" pod "d706291d-60bd-427e-81b2-98523f2f5d2f" (UID: "d706291d-60bd-427e-81b2-98523f2f5d2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:34.417510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.409390 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d706291d-60bd-427e-81b2-98523f2f5d2f-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:34.448866 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.448833 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-maas-api-tls" pod "d706291d-60bd-427e-81b2-98523f2f5d2f" (UID: "d706291d-60bd-427e-81b2-98523f2f5d2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:34.449080 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.448886 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-metrics-certs" pod "d706291d-60bd-427e-81b2-98523f2f5d2f" (UID: "d706291d-60bd-427e-81b2-98523f2f5d2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:34.451394 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.451348 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d706291d-60bd-427e-81b2-98523f2f5d2f-kube-api-access-5crbp" pod "d706291d-60bd-427e-81b2-98523f2f5d2f" (UID: "d706291d-60bd-427e-81b2-98523f2f5d2f"). InnerVolumeSpecName "kube-api-access-5crbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:34.463546 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.463106 5072 generic.go:356] "Generic (PLEG): container finished" podID="d706291d-60bd-427e-81b2-98523f2f5d2f" containerID="d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70" exitCode=0 Oct 30 06:08:34.463546 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.463243 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" event={"ID":"d706291d-60bd-427e-81b2-98523f2f5d2f","Type":"ContainerDied","Data":"d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70"} Oct 30 06:08:34.463546 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.463342 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" event={"ID":"d706291d-60bd-427e-81b2-98523f2f5d2f","Type":"ContainerDied","Data":"b055b0afdb5f3364d44e7985c1650df7c66fc783a455ab5d785cd6172c1a1c01"} Oct 30 06:08:34.463546 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.463370 5072 scope.go:122] "RemoveContainer" containerID="d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70" Oct 30 06:08:34.463895 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.463601 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4" Oct 30 06:08:34.481217 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.481185 5072 scope.go:122] "RemoveContainer" containerID="d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70" Oct 30 06:08:34.481826 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:34.481774 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70\": container with ID starting with d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70 not found: ID does not exist" containerID="d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70" Oct 30 06:08:34.481929 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.481817 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70"} err="failed to get container status \"d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70\": rpc error: code = NotFound desc = could not find container \"d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70\": container with ID starting with d2405d9b3b6a55e2adb9afd3611ec3a9ee83c9615bacec0f1d4ec733ab87ed70 not found: ID does not exist" Oct 30 06:08:34.511385 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.511350 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:34.511385 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.511384 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5crbp\" (UniqueName: \"kubernetes.io/projected/d706291d-60bd-427e-81b2-98523f2f5d2f-kube-api-access-5crbp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:34.511649 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.511401 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d706291d-60bd-427e-81b2-98523f2f5d2f-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:34.527809 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.527772 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4"] Oct 30 06:08:34.527809 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:34.527813 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dbaf6-86bfc5c5fc-qngx4"] Oct 30 06:08:36.521043 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:36.521009 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d706291d-60bd-427e-81b2-98523f2f5d2f" path="/var/lib/kubelet/pods/d706291d-60bd-427e-81b2-98523f2f5d2f/volumes" Oct 30 06:08:42.206979 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:42.206946 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx"] Oct 30 06:08:42.207444 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:42.207161 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" containerID="cri-o://c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8" gracePeriod=30 Oct 30 06:08:44.066491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.064670 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" podUID="ad9e7a96-c07a-4740-9e83-8e2179fff394" containerName="main" containerID="cri-o://b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a" gracePeriod=45 Oct 30 06:08:44.402604 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.402575 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:08:44.521850 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.521819 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-home\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-home\") pod \"ad9e7a96-c07a-4740-9e83-8e2179fff394\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " Oct 30 06:08:44.522157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.521881 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9e7a96-c07a-4740-9e83-8e2179fff394-kube-api-access-stlnh\" (UniqueName: \"kubernetes.io/projected/ad9e7a96-c07a-4740-9e83-8e2179fff394-kube-api-access-stlnh\") pod \"ad9e7a96-c07a-4740-9e83-8e2179fff394\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " Oct 30 06:08:44.522157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.522081 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad9e7a96-c07a-4740-9e83-8e2179fff394-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9e7a96-c07a-4740-9e83-8e2179fff394-tls-certs\") pod \"ad9e7a96-c07a-4740-9e83-8e2179fff394\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " Oct 30 06:08:44.522157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.522128 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-home" pod "ad9e7a96-c07a-4740-9e83-8e2179fff394" (UID: "ad9e7a96-c07a-4740-9e83-8e2179fff394"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:44.522616 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.522592 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-tmp-dir\") pod \"ad9e7a96-c07a-4740-9e83-8e2179fff394\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " Oct 30 06:08:44.522933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.522748 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-dshm\") pod \"ad9e7a96-c07a-4740-9e83-8e2179fff394\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " Oct 30 06:08:44.522933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.522826 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-model-cache\") pod \"ad9e7a96-c07a-4740-9e83-8e2179fff394\" (UID: \"ad9e7a96-c07a-4740-9e83-8e2179fff394\") " Oct 30 06:08:44.523269 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.523113 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-tmp-dir" pod "ad9e7a96-c07a-4740-9e83-8e2179fff394" (UID: "ad9e7a96-c07a-4740-9e83-8e2179fff394"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:44.523363 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.523300 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-model-cache" pod "ad9e7a96-c07a-4740-9e83-8e2179fff394" (UID: "ad9e7a96-c07a-4740-9e83-8e2179fff394"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:44.524073 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.523927 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:44.524073 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.524019 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:44.524073 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.524033 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:44.526253 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.526229 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-dshm" pod "ad9e7a96-c07a-4740-9e83-8e2179fff394" (UID: "ad9e7a96-c07a-4740-9e83-8e2179fff394"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:44.526253 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.526237 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9e7a96-c07a-4740-9e83-8e2179fff394-tls-certs" pod "ad9e7a96-c07a-4740-9e83-8e2179fff394" (UID: "ad9e7a96-c07a-4740-9e83-8e2179fff394"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:44.526422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.526280 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9e7a96-c07a-4740-9e83-8e2179fff394-kube-api-access-stlnh" pod "ad9e7a96-c07a-4740-9e83-8e2179fff394" (UID: "ad9e7a96-c07a-4740-9e83-8e2179fff394"). InnerVolumeSpecName "kube-api-access-stlnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:44.573075 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.573042 5072 generic.go:356] "Generic (PLEG): container finished" podID="ad9e7a96-c07a-4740-9e83-8e2179fff394" containerID="b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a" exitCode=0 Oct 30 06:08:44.573224 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.573092 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" Oct 30 06:08:44.573224 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.573136 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" event={"ID":"ad9e7a96-c07a-4740-9e83-8e2179fff394","Type":"ContainerDied","Data":"b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a"} Oct 30 06:08:44.573224 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.573174 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb" event={"ID":"ad9e7a96-c07a-4740-9e83-8e2179fff394","Type":"ContainerDied","Data":"bb253a8eb6b958881dc1151568384a7e69e539589049fc811fa0b2c7774a8680"} Oct 30 06:08:44.573224 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.573191 5072 scope.go:122] "RemoveContainer" containerID="b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a" Oct 30 06:08:44.584611 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.584589 5072 scope.go:122] "RemoveContainer" containerID="b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a" Oct 30 06:08:44.585168 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:44.585106 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a\": container with ID starting with b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a not found: ID does not exist" containerID="b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a" Oct 30 06:08:44.585268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.585180 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a"} err="failed to get container status \"b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a\": rpc error: code = NotFound desc = could not find container \"b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a\": container with ID starting with b4d735a5c8ac9abec68b8b0b3cabd2f67a40b6844f501caa78546a0c25c8075a not found: ID does not exist" Oct 30 06:08:44.621126 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.621030 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg"] Oct 30 06:08:44.629342 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.629313 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb"] Oct 30 06:08:44.632053 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.632025 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9e7a96-c07a-4740-9e83-8e2179fff394-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:44.632183 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.632058 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlnh\" (UniqueName: \"kubernetes.io/projected/ad9e7a96-c07a-4740-9e83-8e2179fff394-kube-api-access-stlnh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:44.632183 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.632076 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9e7a96-c07a-4740-9e83-8e2179fff394-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:44.634819 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:44.634794 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f28e2d/test-model-f28e2d-kserve-6569db5d77-nptbb"] Oct 30 06:08:46.522606 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:46.522572 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9e7a96-c07a-4740-9e83-8e2179fff394" path="/var/lib/kubelet/pods/ad9e7a96-c07a-4740-9e83-8e2179fff394/volumes" Oct 30 06:08:47.575972 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.575943 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 06:08:47.646076 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.646038 5072 generic.go:356] "Generic (PLEG): container finished" podID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerID="c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8" exitCode=0 Oct 30 06:08:47.646363 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.646342 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" event={"ID":"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a","Type":"ContainerDied","Data":"c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8"} Oct 30 06:08:47.646578 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.646559 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" event={"ID":"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a","Type":"ContainerDied","Data":"e8f7f62cfd2b6651b1186bc296f4472bbc2fab19263c971e6520e63a59f52772"} Oct 30 06:08:47.646721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.646708 5072 scope.go:122] "RemoveContainer" containerID="c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8" Oct 30 06:08:47.647013 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.646996 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx" Oct 30 06:08:47.679094 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.679065 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-podinfo\") pod \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " Oct 30 06:08:47.679244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.679224 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-data\") pod \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " Oct 30 06:08:47.680023 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.679565 5072 scope.go:122] "RemoveContainer" containerID="c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03" Oct 30 06:08:47.680337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.679668 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-data" pod "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" (UID: "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.681734 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.680041 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-token\") pod \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " Oct 30 06:08:47.686804 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.686783 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-kube-api-access-zj2pl\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-kube-api-access-zj2pl\") pod \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " Oct 30 06:08:47.687876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.687858 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-certs\") pod \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " Oct 30 06:08:47.688071 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.688056 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-envoy\") pod \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " Oct 30 06:08:47.688269 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.683940 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-token" pod "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" (UID: "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:47.688362 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.688188 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-certs" pod "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" (UID: "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.688445 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.688426 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istiod-ca-cert\") pod \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " Oct 30 06:08:47.688646 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.688630 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-socket\") pod \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " Oct 30 06:08:47.688869 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.688855 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-credential-socket\") pod \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\" (UID: \"0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a\") " Oct 30 06:08:47.689747 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.689727 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.689867 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.689856 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.689957 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.689946 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.690336 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.690319 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-credential-socket" pod "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" (UID: "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.690668 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.690649 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-socket" pod "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" (UID: "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.690833 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.690808 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istiod-ca-cert" pod "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" (UID: "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:47.693048 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.693017 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-envoy" pod "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" (UID: "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.696845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.696818 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-podinfo" pod "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" (UID: "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:47.698215 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.698182 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-kube-api-access-zj2pl" pod "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" (UID: "0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a"). InnerVolumeSpecName "kube-api-access-zj2pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:47.730452 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.730421 5072 scope.go:122] "RemoveContainer" containerID="c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8" Oct 30 06:08:47.730871 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:47.730841 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8\": container with ID starting with c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8 not found: ID does not exist" containerID="c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8" Oct 30 06:08:47.730972 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.730885 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8"} err="failed to get container status \"c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8\": rpc error: code = NotFound desc = could not find container \"c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8\": container with ID starting with c0f17637e9b1758b9385d4501a5c7a2861ca9319d9327e7705fc86298e635df8 not found: ID does not exist" Oct 30 06:08:47.730972 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.730914 5072 scope.go:122] "RemoveContainer" containerID="c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03" Oct 30 06:08:47.731197 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:08:47.731170 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03\": container with ID starting with c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03 not found: ID does not exist" containerID="c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03" Oct 30 06:08:47.731279 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.731206 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03"} err="failed to get container status \"c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03\": rpc error: code = NotFound desc = could not find container \"c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03\": container with ID starting with c00a321606c0964c8c9c2f52e854b5843bb4445017c61877cd9bb40ab0d44a03 not found: ID does not exist" Oct 30 06:08:47.791519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.791434 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.791519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.791481 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.791519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.791491 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.791519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.791501 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.791519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.791511 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.791519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:47.791520 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj2pl\" (UniqueName: \"kubernetes.io/projected/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a-kube-api-access-zj2pl\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:08:48.007603 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:48.007569 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx"] Oct 30 06:08:48.019439 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:48.019409 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-ldlpx"] Oct 30 06:08:48.524803 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:48.524767 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" path="/var/lib/kubelet/pods/0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a/volumes" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.418414 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh"] Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419200 5072 cpu_manager.go:414] "containerMap: removing container" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419216 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419228 5072 cpu_manager.go:414] "containerMap: removing container" podUID="ad9e7a96-c07a-4740-9e83-8e2179fff394" containerName="main" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419233 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9e7a96-c07a-4740-9e83-8e2179fff394" containerName="main" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419246 5072 cpu_manager.go:414] "containerMap: removing container" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419253 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419271 5072 cpu_manager.go:414] "containerMap: removing container" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419276 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419298 5072 cpu_manager.go:414] "containerMap: removing container" podUID="7ee09dc1-83a8-40af-9a26-7ac225f8506d" containerName="main" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419303 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee09dc1-83a8-40af-9a26-7ac225f8506d" containerName="main" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419311 5072 cpu_manager.go:414] "containerMap: removing container" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419316 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419323 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d66b8be3-8690-4f96-a0e1-223a035ea62e" containerName="maas-api" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419327 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66b8be3-8690-4f96-a0e1-223a035ea62e" containerName="maas-api" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419341 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d706291d-60bd-427e-81b2-98523f2f5d2f" containerName="maas-api" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419345 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d706291d-60bd-427e-81b2-98523f2f5d2f" containerName="maas-api" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419357 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419362 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419376 5072 cpu_manager.go:414] "containerMap: removing container" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419382 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419399 5072 cpu_manager.go:414] "containerMap: removing container" podUID="283adf34-7bce-4a42-9a6a-fbd8009ca69a" containerName="main" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419404 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283adf34-7bce-4a42-9a6a-fbd8009ca69a" containerName="main" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419542 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419553 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9e7a96-c07a-4740-9e83-8e2179fff394" containerName="main" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419562 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419573 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9f028f-843f-4b0b-bdb4-01b3f4b98a9a" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419584 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee09dc1-83a8-40af-9a26-7ac225f8506d" containerName="main" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419593 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffbb6af-8b6b-483c-83df-fc8fe5e9fe72" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419602 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66b8be3-8690-4f96-a0e1-223a035ea62e" containerName="maas-api" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419612 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d706291d-60bd-427e-81b2-98523f2f5d2f" containerName="maas-api" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419619 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f17885-3892-490b-bc8b-8499db744394" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419631 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="715162c8-0a95-4c0d-b15e-3e57c3b707f3" containerName="istio-proxy" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.419643 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="283adf34-7bce-4a42-9a6a-fbd8009ca69a" containerName="main" Oct 30 06:08:52.427111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.426150 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.451818 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.451782 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh"] Oct 30 06:08:52.457651 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.457623 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b8f87\"/\"kube-root-ca.crt\"" Oct 30 06:08:52.457851 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.457827 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b8f87\"/\"openshift-service-ca.crt\"" Oct 30 06:08:52.457998 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.457959 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b8f87\"/\"default-dockercfg-7mscp\"" Oct 30 06:08:52.460085 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.458079 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b8f87\"/\"auth-test-model-6b8f87-kserve-self-signed-certs\"" Oct 30 06:08:52.606117 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.606078 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c5b96f-849c-421a-a843-455581f89c2b-tls-certs\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.606296 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.606268 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zwmv\" (UniqueName: \"kubernetes.io/projected/a0c5b96f-849c-421a-a843-455581f89c2b-kube-api-access-7zwmv\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.606338 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.606301 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-model-cache\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.606419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.606408 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-dshm\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.606498 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.606487 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-tmp-dir\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.606674 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.606650 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-home\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.762524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.750884 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c5b96f-849c-421a-a843-455581f89c2b-tls-certs\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.762962 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.762930 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwmv\" (UniqueName: \"kubernetes.io/projected/a0c5b96f-849c-421a-a843-455581f89c2b-kube-api-access-7zwmv\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.763329 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.763225 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-model-cache\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.763587 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.763568 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-dshm\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.771676 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.765986 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-model-cache\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.772170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.772126 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-tmp-dir\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.772488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.772437 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-home\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.773015 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.772991 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-home\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.773338 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.773302 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-tmp-dir\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.779604 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.774976 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c5b96f-849c-421a-a843-455581f89c2b-tls-certs\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.779604 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.776231 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-dshm\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:52.792187 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:52.792105 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwmv\" (UniqueName: \"kubernetes.io/projected/a0c5b96f-849c-421a-a843-455581f89c2b-kube-api-access-7zwmv\") pod \"auth-test-model-6b8f87-kserve-57fc875657-6pgdh\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:53.053662 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:53.053573 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:53.442499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:53.440037 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh"] Oct 30 06:08:53.830584 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:53.829072 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" event={"ID":"a0c5b96f-849c-421a-a843-455581f89c2b","Type":"ContainerStarted","Data":"84bbc47a1e50caeed4f634f180494c9094dc43182d5d93739b98f90eba17bd1f"} Oct 30 06:08:53.830584 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:53.829119 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" event={"ID":"a0c5b96f-849c-421a-a843-455581f89c2b","Type":"ContainerStarted","Data":"6ceb6283b43954b050bbc7c5b7365bf4180d5393730cc5589c022f007b0ab0d4"} Oct 30 06:08:53.874169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:53.874115 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" podStartSLOduration=1.8740985540000001 podStartE2EDuration="1.874098554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:53.872036345 +0000 UTC m=+2293.475725602" watchObservedRunningTime="2026-10-30 06:08:53.874098554 +0000 UTC m=+2293.477787810" Oct 30 06:08:54.056527 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:54.056481 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:08:59.668484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:59.664980 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" podUID="2d6811e9-6b55-4ec8-bccd-a34c011fe27e" containerName="main" containerID="cri-o://78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345" gracePeriod=45 Oct 30 06:08:59.966035 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:59.963986 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz"] Oct 30 06:08:59.966035 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:08:59.964315 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" containerID="cri-o://fc6c2a4d4657277876509ccad3946bcebcacc21dbf63b7b1b2660e2b711d983d" gracePeriod=30 Oct 30 06:09:00.148173 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.148056 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:09:00.150343 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.150106 5072 generic.go:356] "Generic (PLEG): container finished" podID="2d6811e9-6b55-4ec8-bccd-a34c011fe27e" containerID="78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345" exitCode=0 Oct 30 06:09:00.150854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.150631 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" event={"ID":"2d6811e9-6b55-4ec8-bccd-a34c011fe27e","Type":"ContainerDied","Data":"78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345"} Oct 30 06:09:00.150854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.150679 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" event={"ID":"2d6811e9-6b55-4ec8-bccd-a34c011fe27e","Type":"ContainerDied","Data":"425b87fa513b8fe968b3eedf1697a660066fbab279689019e6288d97765eeb5b"} Oct 30 06:09:00.150854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.150704 5072 scope.go:122] "RemoveContainer" containerID="78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345" Oct 30 06:09:00.221702 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.221263 5072 scope.go:122] "RemoveContainer" containerID="78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345" Oct 30 06:09:00.224303 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:09:00.224268 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345\": container with ID starting with 78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345 not found: ID does not exist" containerID="78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345" Oct 30 06:09:00.224429 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.224314 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345"} err="failed to get container status \"78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345\": rpc error: code = NotFound desc = could not find container \"78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345\": container with ID starting with 78fb081bfb6e0fb71b58a1bc7b925d5a8e6286c0785341683e32071b9c079345 not found: ID does not exist" Oct 30 06:09:00.235374 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.235334 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-kube-api-access-n5jqp\" (UniqueName: \"kubernetes.io/projected/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-kube-api-access-n5jqp\") pod \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " Oct 30 06:09:00.235557 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.235517 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-home\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-home\") pod \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " Oct 30 06:09:00.235692 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.235675 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-model-cache\") pod \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " Oct 30 06:09:00.235773 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.235760 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-dshm\") pod \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " Oct 30 06:09:00.235837 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.235817 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tmp-dir\") pod \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " Oct 30 06:09:00.235905 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.235865 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tls-certs\") pod \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\" (UID: \"2d6811e9-6b55-4ec8-bccd-a34c011fe27e\") " Oct 30 06:09:00.237837 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.237569 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-model-cache" pod "2d6811e9-6b55-4ec8-bccd-a34c011fe27e" (UID: "2d6811e9-6b55-4ec8-bccd-a34c011fe27e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:00.237969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.237896 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-home" pod "2d6811e9-6b55-4ec8-bccd-a34c011fe27e" (UID: "2d6811e9-6b55-4ec8-bccd-a34c011fe27e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:00.239793 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.239746 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tmp-dir" pod "2d6811e9-6b55-4ec8-bccd-a34c011fe27e" (UID: "2d6811e9-6b55-4ec8-bccd-a34c011fe27e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:00.240377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.240349 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tls-certs" pod "2d6811e9-6b55-4ec8-bccd-a34c011fe27e" (UID: "2d6811e9-6b55-4ec8-bccd-a34c011fe27e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:00.240619 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.240592 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-dshm" pod "2d6811e9-6b55-4ec8-bccd-a34c011fe27e" (UID: "2d6811e9-6b55-4ec8-bccd-a34c011fe27e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:00.241625 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.241601 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-kube-api-access-n5jqp" pod "2d6811e9-6b55-4ec8-bccd-a34c011fe27e" (UID: "2d6811e9-6b55-4ec8-bccd-a34c011fe27e"). InnerVolumeSpecName "kube-api-access-n5jqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:00.337764 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.337731 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5jqp\" (UniqueName: \"kubernetes.io/projected/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-kube-api-access-n5jqp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:00.337764 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.337760 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:00.337764 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.337771 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:00.337974 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.337779 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:00.337974 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.337787 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:00.337974 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.337794 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6811e9-6b55-4ec8-bccd-a34c011fe27e-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:00.716054 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:09:00.716002 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:09:00.950478 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:00.948879 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" podUID="2d6811e9-6b55-4ec8-bccd-a34c011fe27e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:09:01.162517 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.162011 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg" Oct 30 06:09:01.202739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.202649 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj"] Oct 30 06:09:01.204953 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.203944 5072 cpu_manager.go:414] "containerMap: removing container" podUID="2d6811e9-6b55-4ec8-bccd-a34c011fe27e" containerName="main" Oct 30 06:09:01.204953 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.204152 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6811e9-6b55-4ec8-bccd-a34c011fe27e" containerName="main" Oct 30 06:09:01.204953 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.204503 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6811e9-6b55-4ec8-bccd-a34c011fe27e" containerName="main" Oct 30 06:09:01.215512 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.210587 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.251516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.234997 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-c2be21-openshift-default-dockercfg-cv24c\"" Oct 30 06:09:01.251516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.250434 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg"] Oct 30 06:09:01.258621 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.258578 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7dfcb7/test-model-7dfcb7-kserve-7945958485-8hlwg"] Oct 30 06:09:01.267795 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.265975 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj"] Oct 30 06:09:01.368483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.361520 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f562x\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-kube-api-access-f562x\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.368483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.361597 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-certs\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.368483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.361637 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-credential-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.368483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.361702 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-envoy\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.368483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.361789 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-podinfo\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.368483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.361912 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-token\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.368483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.361962 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b959f379-69f3-4be5-ac96-8fd4bc2211be-istiod-ca-cert\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.368483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.362014 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-data\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.368483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.362065 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.469390 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-token\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.469495 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b959f379-69f3-4be5-ac96-8fd4bc2211be-istiod-ca-cert\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.469552 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-data\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.469609 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.469668 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f562x\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-kube-api-access-f562x\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.469726 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-certs\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.469818 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-credential-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.469881 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-envoy\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.469974 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-podinfo\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.473739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.470218 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.541157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.514728 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-podinfo\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.541157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.515532 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-certs\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.541157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.516785 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-credential-socket\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.541157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.532983 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-data\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.551497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.549017 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b959f379-69f3-4be5-ac96-8fd4bc2211be-istiod-ca-cert\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.560483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.559717 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-envoy\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.572005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.571687 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-token\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.617542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.609663 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f562x\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-kube-api-access-f562x\") pod \"e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:01.854857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:01.854822 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:02.185215 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:09:02.185177 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb959f379_69f3_4be5_ac96_8fd4bc2211be.slice/crio-e7eb5c8e3ab56e064acc3aad928a8bec2749e6917a1a03cf91b49b35e7129092 WatchSource:0}: Error finding container e7eb5c8e3ab56e064acc3aad928a8bec2749e6917a1a03cf91b49b35e7129092: Status 404 returned error can't find the container with id e7eb5c8e3ab56e064acc3aad928a8bec2749e6917a1a03cf91b49b35e7129092 Oct 30 06:09:02.185668 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:02.185646 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj"] Oct 30 06:09:02.188031 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:02.187996 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:09:02.188143 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:02.188086 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:09:02.188143 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:02.188132 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:09:02.524366 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:02.524278 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6811e9-6b55-4ec8-bccd-a34c011fe27e" path="/var/lib/kubelet/pods/2d6811e9-6b55-4ec8-bccd-a34c011fe27e/volumes" Oct 30 06:09:03.054124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:03.054087 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:09:03.067340 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:03.067310 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:09:03.181171 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:03.181132 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" event={"ID":"b959f379-69f3-4be5-ac96-8fd4bc2211be","Type":"ContainerStarted","Data":"8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d"} Oct 30 06:09:03.181381 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:03.181197 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" event={"ID":"b959f379-69f3-4be5-ac96-8fd4bc2211be","Type":"ContainerStarted","Data":"e7eb5c8e3ab56e064acc3aad928a8bec2749e6917a1a03cf91b49b35e7129092"} Oct 30 06:09:03.193183 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:03.193150 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:09:03.226274 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:03.226213 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" podStartSLOduration=2.2261943730000002 podStartE2EDuration="2.226194373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:03.225274585 +0000 UTC m=+2302.828963843" watchObservedRunningTime="2026-10-30 06:09:03.226194373 +0000 UTC m=+2302.829883632" Oct 30 06:09:03.855821 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:03.855784 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:03.863284 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:03.863258 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:04.191197 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:04.191105 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:04.192037 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:04.192017 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:09:05.204626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.204590 5072 generic.go:356] "Generic (PLEG): container finished" podID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerID="fc6c2a4d4657277876509ccad3946bcebcacc21dbf63b7b1b2660e2b711d983d" exitCode=0 Oct 30 06:09:05.205027 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.204891 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" event={"ID":"52f96c85-f7ff-4335-85e9-2a998e9b8b66","Type":"ContainerDied","Data":"fc6c2a4d4657277876509ccad3946bcebcacc21dbf63b7b1b2660e2b711d983d"} Oct 30 06:09:05.205027 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.204949 5072 scope.go:122] "RemoveContainer" containerID="3f785a16d41b9b6ce314f1ae38ee73256bff6f66e43350b7874df874d55f09f5" Oct 30 06:09:05.305510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.302952 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:09:05.344751 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.344718 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-envoy\") pod \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " Oct 30 06:09:05.344939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.344794 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-data\") pod \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " Oct 30 06:09:05.344939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.344843 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-certs\") pod \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " Oct 30 06:09:05.344939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.344913 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-podinfo\") pod \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " Oct 30 06:09:05.345052 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.345011 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-socket\") pod \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " Oct 30 06:09:05.345124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.345108 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-token\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-token\") pod \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " Oct 30 06:09:05.345218 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.345204 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-credential-socket\") pod \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " Oct 30 06:09:05.345294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.345280 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istiod-ca-cert\") pod \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " Oct 30 06:09:05.345336 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.345319 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-kube-api-access-hjl45\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-kube-api-access-hjl45\") pod \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\" (UID: \"52f96c85-f7ff-4335-85e9-2a998e9b8b66\") " Oct 30 06:09:05.345512 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.345106 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-data" pod "52f96c85-f7ff-4335-85e9-2a998e9b8b66" (UID: "52f96c85-f7ff-4335-85e9-2a998e9b8b66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.346275 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.345623 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-certs" pod "52f96c85-f7ff-4335-85e9-2a998e9b8b66" (UID: "52f96c85-f7ff-4335-85e9-2a998e9b8b66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.346832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.345714 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-socket" pod "52f96c85-f7ff-4335-85e9-2a998e9b8b66" (UID: "52f96c85-f7ff-4335-85e9-2a998e9b8b66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.346832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.345731 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-credential-socket" pod "52f96c85-f7ff-4335-85e9-2a998e9b8b66" (UID: "52f96c85-f7ff-4335-85e9-2a998e9b8b66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.346832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.346061 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istiod-ca-cert" pod "52f96c85-f7ff-4335-85e9-2a998e9b8b66" (UID: "52f96c85-f7ff-4335-85e9-2a998e9b8b66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:05.346832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.346779 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.346832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.346792 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.346832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.346804 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.346832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.346813 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.348044 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.348020 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-podinfo" pod "52f96c85-f7ff-4335-85e9-2a998e9b8b66" (UID: "52f96c85-f7ff-4335-85e9-2a998e9b8b66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:05.348445 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.348421 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-kube-api-access-hjl45" pod "52f96c85-f7ff-4335-85e9-2a998e9b8b66" (UID: "52f96c85-f7ff-4335-85e9-2a998e9b8b66"). InnerVolumeSpecName "kube-api-access-hjl45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:05.348722 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.348695 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-token" pod "52f96c85-f7ff-4335-85e9-2a998e9b8b66" (UID: "52f96c85-f7ff-4335-85e9-2a998e9b8b66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:05.349171 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.349146 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-envoy" pod "52f96c85-f7ff-4335-85e9-2a998e9b8b66" (UID: "52f96c85-f7ff-4335-85e9-2a998e9b8b66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.457330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.454558 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjl45\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-kube-api-access-hjl45\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.457330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.454597 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.457330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.454612 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f96c85-f7ff-4335-85e9-2a998e9b8b66-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.457330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.454627 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.457330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:05.454640 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f96c85-f7ff-4335-85e9-2a998e9b8b66-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:06.214063 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:06.214022 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" Oct 30 06:09:06.214618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:06.214028 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz" event={"ID":"52f96c85-f7ff-4335-85e9-2a998e9b8b66","Type":"ContainerDied","Data":"25b244d4a7d6d97d64ca2870a9365027d3026c2d3693155adfe5c5c46ef868ce"} Oct 30 06:09:06.214618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:06.214213 5072 scope.go:122] "RemoveContainer" containerID="fc6c2a4d4657277876509ccad3946bcebcacc21dbf63b7b1b2660e2b711d983d" Oct 30 06:09:06.286267 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:06.286232 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz"] Oct 30 06:09:06.299775 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:06.299740 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-hrxkz"] Oct 30 06:09:06.522155 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:06.522079 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" path="/var/lib/kubelet/pods/52f96c85-f7ff-4335-85e9-2a998e9b8b66/volumes" Oct 30 06:09:07.949477 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:07.949116 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf"] Oct 30 06:09:07.949952 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:07.949832 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" containerID="cri-o://e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876" gracePeriod=30 Oct 30 06:09:10.646237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:10.646201 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74"] Oct 30 06:09:10.647202 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:10.647143 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" podUID="b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" containerName="maas-api" containerID="cri-o://f92cad136c4a36e6d92312fde07d0dd3fb052f93ab4be36e8729f4c33b14f0d3" gracePeriod=30 Oct 30 06:09:11.280992 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.280957 5072 generic.go:356] "Generic (PLEG): container finished" podID="b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" containerID="f92cad136c4a36e6d92312fde07d0dd3fb052f93ab4be36e8729f4c33b14f0d3" exitCode=0 Oct 30 06:09:11.281217 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.281195 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" event={"ID":"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a","Type":"ContainerDied","Data":"f92cad136c4a36e6d92312fde07d0dd3fb052f93ab4be36e8729f4c33b14f0d3"} Oct 30 06:09:11.319164 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.319132 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:09:11.413623 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.413585 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-kube-api-access-m4tc6\" (UniqueName: \"kubernetes.io/projected/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-kube-api-access-m4tc6\") pod \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " Oct 30 06:09:11.413844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.413649 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-openshift-service-ca\") pod \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " Oct 30 06:09:11.413844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.413839 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-maas-api-tls\") pod \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " Oct 30 06:09:11.413966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.413955 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-metrics-certs\") pod \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\" (UID: \"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a\") " Oct 30 06:09:11.414142 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.414107 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-openshift-service-ca" pod "b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" (UID: "b1a96066-f1b8-4c2b-964d-54fbf3f7f41a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:11.414718 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.414695 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.416978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.416948 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-maas-api-tls" pod "b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" (UID: "b1a96066-f1b8-4c2b-964d-54fbf3f7f41a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:11.416978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.416965 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-kube-api-access-m4tc6" pod "b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" (UID: "b1a96066-f1b8-4c2b-964d-54fbf3f7f41a"). InnerVolumeSpecName "kube-api-access-m4tc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:11.417904 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.417883 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-metrics-certs" pod "b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" (UID: "b1a96066-f1b8-4c2b-964d-54fbf3f7f41a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:11.515969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.515933 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4tc6\" (UniqueName: \"kubernetes.io/projected/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-kube-api-access-m4tc6\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.515969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.515964 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.515969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:11.515975 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.290343 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:12.290313 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" Oct 30 06:09:12.290845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:12.290350 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74" event={"ID":"b1a96066-f1b8-4c2b-964d-54fbf3f7f41a","Type":"ContainerDied","Data":"334508b55cdcce0bb3f1c4e76b6407e060b5b2773d07e01339219f4b20889e30"} Oct 30 06:09:12.290845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:12.290427 5072 scope.go:122] "RemoveContainer" containerID="f92cad136c4a36e6d92312fde07d0dd3fb052f93ab4be36e8729f4c33b14f0d3" Oct 30 06:09:12.345169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:12.345134 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74"] Oct 30 06:09:12.354974 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:12.354939 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-482468-cb99f4dcc-4dz74"] Oct 30 06:09:12.520606 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:12.520570 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" path="/var/lib/kubelet/pods/b1a96066-f1b8-4c2b-964d-54fbf3f7f41a/volumes" Oct 30 06:09:13.249290 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.249263 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 06:09:13.301089 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.301057 5072 generic.go:356] "Generic (PLEG): container finished" podID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerID="e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876" exitCode=0 Oct 30 06:09:13.301590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.301126 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" Oct 30 06:09:13.301590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.301145 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" event={"ID":"ba5710a2-f577-4d8a-b42e-48dbf864430f","Type":"ContainerDied","Data":"e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876"} Oct 30 06:09:13.301590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.301218 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf" event={"ID":"ba5710a2-f577-4d8a-b42e-48dbf864430f","Type":"ContainerDied","Data":"0db6bdaf88b607fc50e6c054690616d0ecbeeb0a2136bc4cdfeea86ef527b342"} Oct 30 06:09:13.301590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.301250 5072 scope.go:122] "RemoveContainer" containerID="e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876" Oct 30 06:09:13.329171 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.328876 5072 scope.go:122] "RemoveContainer" containerID="e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876" Oct 30 06:09:13.329520 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:09:13.329421 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876\": container with ID starting with e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876 not found: ID does not exist" containerID="e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876" Oct 30 06:09:13.329520 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.329478 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876"} err="failed to get container status \"e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876\": rpc error: code = NotFound desc = could not find container \"e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876\": container with ID starting with e400b0a7f1e58d73657504bb16f2bf25bd7390963291269471f60041b9361876 not found: ID does not exist" Oct 30 06:09:13.345115 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.345080 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-certs\") pod \"ba5710a2-f577-4d8a-b42e-48dbf864430f\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " Oct 30 06:09:13.345267 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.345174 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-socket\") pod \"ba5710a2-f577-4d8a-b42e-48dbf864430f\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " Oct 30 06:09:13.345267 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.345217 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-token\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-token\") pod \"ba5710a2-f577-4d8a-b42e-48dbf864430f\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " Oct 30 06:09:13.345436 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.345415 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5710a2-f577-4d8a-b42e-48dbf864430f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5710a2-f577-4d8a-b42e-48dbf864430f-istiod-ca-cert\") pod \"ba5710a2-f577-4d8a-b42e-48dbf864430f\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " Oct 30 06:09:13.345547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.345472 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-credential-socket\") pod \"ba5710a2-f577-4d8a-b42e-48dbf864430f\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " Oct 30 06:09:13.345547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.345531 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-envoy\") pod \"ba5710a2-f577-4d8a-b42e-48dbf864430f\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " Oct 30 06:09:13.345651 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.345582 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-data\") pod \"ba5710a2-f577-4d8a-b42e-48dbf864430f\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " Oct 30 06:09:13.345711 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.345664 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-podinfo\") pod \"ba5710a2-f577-4d8a-b42e-48dbf864430f\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " Oct 30 06:09:13.345771 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.345758 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-kube-api-access-2t6v9\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-kube-api-access-2t6v9\") pod \"ba5710a2-f577-4d8a-b42e-48dbf864430f\" (UID: \"ba5710a2-f577-4d8a-b42e-48dbf864430f\") " Oct 30 06:09:13.346076 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.346052 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-certs" pod "ba5710a2-f577-4d8a-b42e-48dbf864430f" (UID: "ba5710a2-f577-4d8a-b42e-48dbf864430f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.346675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.346655 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.346792 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.346703 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-credential-socket" pod "ba5710a2-f577-4d8a-b42e-48dbf864430f" (UID: "ba5710a2-f577-4d8a-b42e-48dbf864430f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.346902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.346876 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-socket" pod "ba5710a2-f577-4d8a-b42e-48dbf864430f" (UID: "ba5710a2-f577-4d8a-b42e-48dbf864430f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.347137 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.347121 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-data" pod "ba5710a2-f577-4d8a-b42e-48dbf864430f" (UID: "ba5710a2-f577-4d8a-b42e-48dbf864430f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.350222 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.348845 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5710a2-f577-4d8a-b42e-48dbf864430f-istiod-ca-cert" pod "ba5710a2-f577-4d8a-b42e-48dbf864430f" (UID: "ba5710a2-f577-4d8a-b42e-48dbf864430f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:13.351322 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.351300 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-podinfo" pod "ba5710a2-f577-4d8a-b42e-48dbf864430f" (UID: "ba5710a2-f577-4d8a-b42e-48dbf864430f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:13.351419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.351375 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-envoy" pod "ba5710a2-f577-4d8a-b42e-48dbf864430f" (UID: "ba5710a2-f577-4d8a-b42e-48dbf864430f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.353221 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.353170 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-token" pod "ba5710a2-f577-4d8a-b42e-48dbf864430f" (UID: "ba5710a2-f577-4d8a-b42e-48dbf864430f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:13.353221 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.353202 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-kube-api-access-2t6v9" pod "ba5710a2-f577-4d8a-b42e-48dbf864430f" (UID: "ba5710a2-f577-4d8a-b42e-48dbf864430f"). InnerVolumeSpecName "kube-api-access-2t6v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:13.448311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.448267 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5710a2-f577-4d8a-b42e-48dbf864430f-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.448311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.448302 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.448311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.448316 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.448591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.448329 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.448591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.448340 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.448591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.448352 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t6v9\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-kube-api-access-2t6v9\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.448591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.448364 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5710a2-f577-4d8a-b42e-48dbf864430f-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.448591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.448378 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5710a2-f577-4d8a-b42e-48dbf864430f-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.651106 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.651069 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf"] Oct 30 06:09:13.658596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:13.658562 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-kz9qf"] Oct 30 06:09:14.522236 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:14.522199 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" path="/var/lib/kubelet/pods/ba5710a2-f577-4d8a-b42e-48dbf864430f/volumes" Oct 30 06:09:18.582751 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:18.582718 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz"] Oct 30 06:09:18.583348 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:18.582986 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" containerID="cri-o://8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b" gracePeriod=30 Oct 30 06:09:20.826499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:20.826427 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl"] Oct 30 06:09:20.827047 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:20.826712 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" podUID="fe7c0d1e-a4c9-4943-9454-c09687532703" containerName="maas-api" containerID="cri-o://77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f" gracePeriod=30 Oct 30 06:09:21.120619 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.120592 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:09:21.188418 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.188386 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls\") pod \"fe7c0d1e-a4c9-4943-9454-c09687532703\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " Oct 30 06:09:21.188637 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.188618 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-metrics-certs\") pod \"fe7c0d1e-a4c9-4943-9454-c09687532703\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " Oct 30 06:09:21.188756 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.188738 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7c0d1e-a4c9-4943-9454-c09687532703-kube-api-access-c67t7\" (UniqueName: \"kubernetes.io/projected/fe7c0d1e-a4c9-4943-9454-c09687532703-kube-api-access-c67t7\") pod \"fe7c0d1e-a4c9-4943-9454-c09687532703\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " Oct 30 06:09:21.188843 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.188811 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe7c0d1e-a4c9-4943-9454-c09687532703-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7c0d1e-a4c9-4943-9454-c09687532703-openshift-service-ca\") pod \"fe7c0d1e-a4c9-4943-9454-c09687532703\" (UID: \"fe7c0d1e-a4c9-4943-9454-c09687532703\") " Oct 30 06:09:21.189696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.189657 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7c0d1e-a4c9-4943-9454-c09687532703-openshift-service-ca" pod "fe7c0d1e-a4c9-4943-9454-c09687532703" (UID: "fe7c0d1e-a4c9-4943-9454-c09687532703"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:21.193129 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.193087 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7c0d1e-a4c9-4943-9454-c09687532703-kube-api-access-c67t7" pod "fe7c0d1e-a4c9-4943-9454-c09687532703" (UID: "fe7c0d1e-a4c9-4943-9454-c09687532703"). InnerVolumeSpecName "kube-api-access-c67t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:21.195580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.195552 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls" pod "fe7c0d1e-a4c9-4943-9454-c09687532703" (UID: "fe7c0d1e-a4c9-4943-9454-c09687532703"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:21.195989 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.195961 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-metrics-certs" pod "fe7c0d1e-a4c9-4943-9454-c09687532703" (UID: "fe7c0d1e-a4c9-4943-9454-c09687532703"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:21.234202 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.234164 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v"] Oct 30 06:09:21.234410 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.234387 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" podUID="6cb02ddc-9823-4971-aaa4-b9d98cf4d857" containerName="maas-api" containerID="cri-o://881b733658374f0c68bd6742af1b63f6da43cade9e891ea3f84d516ab68c6d6e" gracePeriod=30 Oct 30 06:09:21.291249 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.291209 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:21.291249 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.291248 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c67t7\" (UniqueName: \"kubernetes.io/projected/fe7c0d1e-a4c9-4943-9454-c09687532703-kube-api-access-c67t7\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:21.291500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.291263 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe7c0d1e-a4c9-4943-9454-c09687532703-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:21.291500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.291276 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe7c0d1e-a4c9-4943-9454-c09687532703-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:21.396290 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.396257 5072 generic.go:356] "Generic (PLEG): container finished" podID="6cb02ddc-9823-4971-aaa4-b9d98cf4d857" containerID="881b733658374f0c68bd6742af1b63f6da43cade9e891ea3f84d516ab68c6d6e" exitCode=0 Oct 30 06:09:21.396586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.396335 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" event={"ID":"6cb02ddc-9823-4971-aaa4-b9d98cf4d857","Type":"ContainerDied","Data":"881b733658374f0c68bd6742af1b63f6da43cade9e891ea3f84d516ab68c6d6e"} Oct 30 06:09:21.398049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.398020 5072 generic.go:356] "Generic (PLEG): container finished" podID="fe7c0d1e-a4c9-4943-9454-c09687532703" containerID="77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f" exitCode=0 Oct 30 06:09:21.398203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.398082 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" event={"ID":"fe7c0d1e-a4c9-4943-9454-c09687532703","Type":"ContainerDied","Data":"77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f"} Oct 30 06:09:21.398203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.398140 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" Oct 30 06:09:21.398203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.398149 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl" event={"ID":"fe7c0d1e-a4c9-4943-9454-c09687532703","Type":"ContainerDied","Data":"74b300f58a9bc1d5b6f0d2670cfa88639659b61961f13ab0d4d1d9c5d79ed2c6"} Oct 30 06:09:21.398203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.398181 5072 scope.go:122] "RemoveContainer" containerID="77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f" Oct 30 06:09:21.411427 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.411406 5072 scope.go:122] "RemoveContainer" containerID="77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f" Oct 30 06:09:21.411750 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:09:21.411730 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f\": container with ID starting with 77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f not found: ID does not exist" containerID="77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f" Oct 30 06:09:21.411831 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.411760 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f"} err="failed to get container status \"77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f\": rpc error: code = NotFound desc = could not find container \"77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f\": container with ID starting with 77584d595ecd7f6cb9a19bc72a718217be697373b7d6346e4b14c96a8f930d7f not found: ID does not exist" Oct 30 06:09:21.465831 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.465780 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl"] Oct 30 06:09:21.524724 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.518724 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:09:21.616582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.614353 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls\") pod \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " Oct 30 06:09:21.616582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.614403 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-metrics-certs\") pod \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " Oct 30 06:09:21.616582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.614524 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-kube-api-access-w8bt8\" (UniqueName: \"kubernetes.io/projected/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-kube-api-access-w8bt8\") pod \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " Oct 30 06:09:21.616582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.614593 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-openshift-service-ca\") pod \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\" (UID: \"6cb02ddc-9823-4971-aaa4-b9d98cf4d857\") " Oct 30 06:09:21.633811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.633349 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-openshift-service-ca" pod "6cb02ddc-9823-4971-aaa4-b9d98cf4d857" (UID: "6cb02ddc-9823-4971-aaa4-b9d98cf4d857"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:21.634629 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.634602 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls" pod "6cb02ddc-9823-4971-aaa4-b9d98cf4d857" (UID: "6cb02ddc-9823-4971-aaa4-b9d98cf4d857"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:21.636128 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.636102 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-metrics-certs" pod "6cb02ddc-9823-4971-aaa4-b9d98cf4d857" (UID: "6cb02ddc-9823-4971-aaa4-b9d98cf4d857"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:21.642633 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.642590 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-kube-api-access-w8bt8" pod "6cb02ddc-9823-4971-aaa4-b9d98cf4d857" (UID: "6cb02ddc-9823-4971-aaa4-b9d98cf4d857"). InnerVolumeSpecName "kube-api-access-w8bt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:21.643743 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.643504 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f28e2d-5d7c78df5b-z7tnl"] Oct 30 06:09:21.718522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.718074 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:21.718522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.718103 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:21.718522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.718116 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8bt8\" (UniqueName: \"kubernetes.io/projected/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-kube-api-access-w8bt8\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:21.718522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.718131 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb02ddc-9823-4971-aaa4-b9d98cf4d857-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:21.848812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.845671 5072 patch_prober.go:29] interesting pod/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:09:21.848812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:21.845732 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:09:22.407805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:22.407777 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" Oct 30 06:09:22.408010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:22.407802 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v" event={"ID":"6cb02ddc-9823-4971-aaa4-b9d98cf4d857","Type":"ContainerDied","Data":"cbf35b22811a9861075ab4a602ec255c2c08c9206c054a165e5b56c74669bdae"} Oct 30 06:09:22.408010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:22.407862 5072 scope.go:122] "RemoveContainer" containerID="881b733658374f0c68bd6742af1b63f6da43cade9e891ea3f84d516ab68c6d6e" Oct 30 06:09:22.467689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:22.467652 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v"] Oct 30 06:09:22.486648 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:22.486591 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-38cfad-6458cbdfbf-2qv9v"] Oct 30 06:09:22.530850 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:22.530817 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb02ddc-9823-4971-aaa4-b9d98cf4d857" path="/var/lib/kubelet/pods/6cb02ddc-9823-4971-aaa4-b9d98cf4d857/volumes" Oct 30 06:09:22.531600 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:22.531578 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe7c0d1e-a4c9-4943-9454-c09687532703" path="/var/lib/kubelet/pods/fe7c0d1e-a4c9-4943-9454-c09687532703/volumes" Oct 30 06:09:23.889528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.889502 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 06:09:23.960351 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960321 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-podinfo\") pod \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " Oct 30 06:09:23.960571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960390 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-socket\") pod \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " Oct 30 06:09:23.960571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960424 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-token\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-token\") pod \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " Oct 30 06:09:23.960571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960483 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-kube-api-access-frk2c\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-kube-api-access-frk2c\") pod \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " Oct 30 06:09:23.960766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960575 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-envoy\") pod \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " Oct 30 06:09:23.960766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960601 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-credential-socket\") pod \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " Oct 30 06:09:23.960766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960719 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-certs\") pod \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " Oct 30 06:09:23.960766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960757 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-data\") pod \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " Oct 30 06:09:23.960988 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960766 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-socket" pod "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" (UID: "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:23.960988 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960896 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istiod-ca-cert\") pod \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\" (UID: \"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c\") " Oct 30 06:09:23.961098 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.960993 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-credential-socket" pod "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" (UID: "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:23.961207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.961190 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-certs" pod "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" (UID: "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:23.961416 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.961402 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istiod-ca-cert" pod "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" (UID: "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:23.961611 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.961586 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-data" pod "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" (UID: "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:23.961771 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.961746 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:23.962059 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.961774 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:23.962059 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.961790 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:23.962059 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.961805 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:23.962059 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.961818 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:23.963883 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.963232 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-token" pod "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" (UID: "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:23.963883 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.963710 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-envoy" pod "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" (UID: "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:23.964415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.964394 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-kube-api-access-frk2c" pod "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" (UID: "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c"). InnerVolumeSpecName "kube-api-access-frk2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:23.981631 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:23.981603 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-podinfo" pod "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" (UID: "b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:24.063419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.063389 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:24.063419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.063421 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:24.063653 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.063434 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frk2c\" (UniqueName: \"kubernetes.io/projected/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-kube-api-access-frk2c\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:24.063653 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.063443 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:24.429018 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.428950 5072 generic.go:356] "Generic (PLEG): container finished" podID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerID="8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b" exitCode=0 Oct 30 06:09:24.429234 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.429064 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" Oct 30 06:09:24.429234 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.429054 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" event={"ID":"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c","Type":"ContainerDied","Data":"8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b"} Oct 30 06:09:24.429381 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.429239 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz" event={"ID":"b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c","Type":"ContainerDied","Data":"fbc495f341ebae21592d2e21a1ecf3387f4a86dd7fb7652762c0636597c147da"} Oct 30 06:09:24.429381 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.429276 5072 scope.go:122] "RemoveContainer" containerID="8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b" Oct 30 06:09:24.442479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.442439 5072 scope.go:122] "RemoveContainer" containerID="5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234" Oct 30 06:09:24.454652 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.454630 5072 scope.go:122] "RemoveContainer" containerID="8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b" Oct 30 06:09:24.454983 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:09:24.454961 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b\": container with ID starting with 8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b not found: ID does not exist" containerID="8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b" Oct 30 06:09:24.455087 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.454994 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b"} err="failed to get container status \"8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b\": rpc error: code = NotFound desc = could not find container \"8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b\": container with ID starting with 8af8d32d2f2dcf28050390ce8ac0e931c6b47433c9f463249a291fda4e3bb34b not found: ID does not exist" Oct 30 06:09:24.455087 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.455019 5072 scope.go:122] "RemoveContainer" containerID="5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234" Oct 30 06:09:24.455352 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:09:24.455330 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234\": container with ID starting with 5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234 not found: ID does not exist" containerID="5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234" Oct 30 06:09:24.455406 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.455363 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234"} err="failed to get container status \"5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234\": rpc error: code = NotFound desc = could not find container \"5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234\": container with ID starting with 5e1d81123738a72834df34c52a8213c6839ec9868bfec33873f70a3bd6a9c234 not found: ID does not exist" Oct 30 06:09:24.533843 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.533811 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz"] Oct 30 06:09:24.533843 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.533851 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-482468-openshift-default-77c4499f99-nsdvz"] Oct 30 06:09:24.642657 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.642623 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm"] Oct 30 06:09:24.643472 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643435 5072 cpu_manager.go:414] "containerMap: removing container" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" Oct 30 06:09:24.643472 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643474 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643503 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643509 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643517 5072 cpu_manager.go:414] "containerMap: removing container" podUID="fe7c0d1e-a4c9-4943-9454-c09687532703" containerName="maas-api" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643523 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7c0d1e-a4c9-4943-9454-c09687532703" containerName="maas-api" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643547 5072 cpu_manager.go:414] "containerMap: removing container" podUID="6cb02ddc-9823-4971-aaa4-b9d98cf4d857" containerName="maas-api" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643552 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb02ddc-9823-4971-aaa4-b9d98cf4d857" containerName="maas-api" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643560 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" containerName="maas-api" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643565 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" containerName="maas-api" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643572 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643577 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643639 5072 cpu_manager.go:414] "containerMap: removing container" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" Oct 30 06:09:24.643681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643646 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" Oct 30 06:09:24.644386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643808 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb02ddc-9823-4971-aaa4-b9d98cf4d857" containerName="maas-api" Oct 30 06:09:24.644386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643819 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a96066-f1b8-4c2b-964d-54fbf3f7f41a" containerName="maas-api" Oct 30 06:09:24.644386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643841 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" Oct 30 06:09:24.644386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643855 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" Oct 30 06:09:24.644386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643865 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5710a2-f577-4d8a-b42e-48dbf864430f" containerName="istio-proxy" Oct 30 06:09:24.644386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643876 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" Oct 30 06:09:24.644386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643886 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" containerName="istio-proxy" Oct 30 06:09:24.644386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.643898 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7c0d1e-a4c9-4943-9454-c09687532703" containerName="maas-api" Oct 30 06:09:24.649769 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.649742 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.664859 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.664827 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-02ceed\"/\"openshift-service-ca.crt\"" Oct 30 06:09:24.665716 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.665689 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-02ceed\"/\"default-dockercfg-s5wrt\"" Oct 30 06:09:24.665881 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.665861 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-02ceed\"/\"rate-test-model-02ceed-kserve-self-signed-certs\"" Oct 30 06:09:24.665973 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.665957 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-02ceed\"/\"kube-root-ca.crt\"" Oct 30 06:09:24.668143 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.668116 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm"] Oct 30 06:09:24.788669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.788573 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47e16b1-6b33-43bf-804f-865ab51826df-tls-certs\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.788852 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.788758 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/f47e16b1-6b33-43bf-804f-865ab51826df-kube-api-access-s5wmz\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.788913 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.788883 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-model-cache\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.788957 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.788946 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-dshm\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.789013 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.788976 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-tmp-dir\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.789108 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.789091 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-home\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.916484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.914943 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-tmp-dir\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.916484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.915030 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-home\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.916484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.915119 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47e16b1-6b33-43bf-804f-865ab51826df-tls-certs\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.916484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.915220 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/f47e16b1-6b33-43bf-804f-865ab51826df-kube-api-access-s5wmz\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.916484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.915317 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-model-cache\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.916484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.915402 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-dshm\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.929484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.928989 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-tmp-dir\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.929484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.929353 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-model-cache\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.929734 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.929630 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-home\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.956548 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.952762 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-dshm\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.957826 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.957257 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47e16b1-6b33-43bf-804f-865ab51826df-tls-certs\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.977784 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.977246 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/f47e16b1-6b33-43bf-804f-865ab51826df-kube-api-access-s5wmz\") pod \"rate-test-model-02ceed-kserve-79869fb589-zsptm\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:24.977784 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:24.977559 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:25.336815 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:09:25.336778 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf47e16b1_6b33_43bf_804f_865ab51826df.slice/crio-a9f15664a0ee6f469de1367cccc417f6fbd005309a5ba252dece696a2ddc1f46 WatchSource:0}: Error finding container a9f15664a0ee6f469de1367cccc417f6fbd005309a5ba252dece696a2ddc1f46: Status 404 returned error can't find the container with id a9f15664a0ee6f469de1367cccc417f6fbd005309a5ba252dece696a2ddc1f46 Oct 30 06:09:25.339674 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.339648 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm"] Oct 30 06:09:25.440497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.440419 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" event={"ID":"f47e16b1-6b33-43bf-804f-865ab51826df","Type":"ContainerStarted","Data":"087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587"} Oct 30 06:09:25.440497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.440491 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" event={"ID":"f47e16b1-6b33-43bf-804f-865ab51826df","Type":"ContainerStarted","Data":"a9f15664a0ee6f469de1367cccc417f6fbd005309a5ba252dece696a2ddc1f46"} Oct 30 06:09:25.486869 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.486796 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" podStartSLOduration=1.486775393 podStartE2EDuration="1.486775393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:25.483177814 +0000 UTC m=+2325.086867070" watchObservedRunningTime="2026-10-30 06:09:25.486775393 +0000 UTC m=+2325.090464650" Oct 30 06:09:25.499226 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.499193 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq"] Oct 30 06:09:25.500040 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.500012 5072 cpu_manager.go:414] "containerMap: removing container" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" Oct 30 06:09:25.500040 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.500035 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f96c85-f7ff-4335-85e9-2a998e9b8b66" containerName="istio-proxy" Oct 30 06:09:25.503952 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.503931 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.520586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.520557 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq"] Oct 30 06:09:25.540655 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.520744 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-452a59-openshift-default-dockercfg-wgvvb\"" Oct 30 06:09:25.540871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.539958 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.541021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.541004 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.541258 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.541235 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.541400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.541381 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.541775 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.541544 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.541775 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.541601 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.541775 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.541685 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.541775 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.541715 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.542089 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.541816 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8jw\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-kube-api-access-cv8jw\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.643618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.643583 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.643618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.643620 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.643858 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.643651 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8jw\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-kube-api-access-cv8jw\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.643858 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.643715 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.643858 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.643743 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.643858 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.643819 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.644065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.643862 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.644065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.643968 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.644065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.644022 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.644065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.644047 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.644065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.644065 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.644389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.644346 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.644471 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.644386 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.644682 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.644659 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.646629 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.646607 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.646937 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.646919 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.667039 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.667010 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.667326 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.667301 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8jw\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-kube-api-access-cv8jw\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.846064 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.846028 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:25.978996 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:25.978960 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:26.179444 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:26.179411 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq"] Oct 30 06:09:26.180878 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:09:26.180842 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb930a6_978c_413f_b1fb_6b7ab90d5fc0.slice/crio-dda6def25d63fafa5d655329780bd2a440c533626c3c3267bef1d69447bce393 WatchSource:0}: Error finding container dda6def25d63fafa5d655329780bd2a440c533626c3c3267bef1d69447bce393: Status 404 returned error can't find the container with id dda6def25d63fafa5d655329780bd2a440c533626c3c3267bef1d69447bce393 Oct 30 06:09:26.184758 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:26.184709 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:09:26.184881 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:26.184794 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:09:26.184881 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:26.184831 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:09:26.465693 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:26.465598 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" event={"ID":"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0","Type":"ContainerStarted","Data":"41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406"} Oct 30 06:09:26.465868 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:26.465709 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" event={"ID":"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0","Type":"ContainerStarted","Data":"dda6def25d63fafa5d655329780bd2a440c533626c3c3267bef1d69447bce393"} Oct 30 06:09:26.508267 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:26.508210 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" podStartSLOduration=1.5081953590000001 podStartE2EDuration="1.508195359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:26.50684324 +0000 UTC m=+2326.110532521" watchObservedRunningTime="2026-10-30 06:09:26.508195359 +0000 UTC m=+2326.111884665" Oct 30 06:09:26.521779 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:26.521747 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c" path="/var/lib/kubelet/pods/b19abbb8-25a8-4b8a-bb23-8c1fcda40c1c/volumes" Oct 30 06:09:26.846210 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:26.846123 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:27.848755 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:27.848715 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:09:27.849245 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:27.848782 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 06:09:28.846991 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:28.846940 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:09:28.847192 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:28.847009 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 06:09:29.266560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:29.264424 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f"] Oct 30 06:09:29.266560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:29.265501 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" containerID="cri-o://0c62c319a143899c3d2757e8b8d16071ef59d7240b40533e9c7f88cb9ca8f489" gracePeriod=30 Oct 30 06:09:29.850498 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:29.848142 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:09:29.850498 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:29.848215 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 06:09:30.798485 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:09:30.794270 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:09:30.853337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:30.852870 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:09:30.853337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:30.852930 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 06:09:31.846624 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:31.846584 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:09:31.847094 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:31.846660 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 06:09:33.077378 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:33.077334 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:33.078692 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:33.078669 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:33.079594 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:33.079576 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:09:34.589860 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.589825 5072 generic.go:356] "Generic (PLEG): container finished" podID="0e614f11-3285-4826-aec4-ffa24b359348" containerID="0c62c319a143899c3d2757e8b8d16071ef59d7240b40533e9c7f88cb9ca8f489" exitCode=0 Oct 30 06:09:34.590812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.590786 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" event={"ID":"0e614f11-3285-4826-aec4-ffa24b359348","Type":"ContainerDied","Data":"0c62c319a143899c3d2757e8b8d16071ef59d7240b40533e9c7f88cb9ca8f489"} Oct 30 06:09:34.591025 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.590996 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" event={"ID":"0e614f11-3285-4826-aec4-ffa24b359348","Type":"ContainerDied","Data":"f0ba67e1312dfa1ff460323c73c28b2b153b8e8e1fe97bcb799852bd8fc888ef"} Oct 30 06:09:34.591025 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.591036 5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0ba67e1312dfa1ff460323c73c28b2b153b8e8e1fe97bcb799852bd8fc888ef" Oct 30 06:09:34.591220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.591050 5072 scope.go:122] "RemoveContainer" containerID="15e9fb0be3f0f9847ff14b84421814090d645fcacdbca5206c521b23db44e68f" Oct 30 06:09:34.605016 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.604995 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 06:09:34.748635 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.748597 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-kube-api-access-fspmh\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-kube-api-access-fspmh\") pod \"0e614f11-3285-4826-aec4-ffa24b359348\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " Oct 30 06:09:34.748814 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.748656 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-certs\") pod \"0e614f11-3285-4826-aec4-ffa24b359348\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " Oct 30 06:09:34.748814 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.748687 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-socket\") pod \"0e614f11-3285-4826-aec4-ffa24b359348\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " Oct 30 06:09:34.748814 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.748809 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-credential-socket\") pod \"0e614f11-3285-4826-aec4-ffa24b359348\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " Oct 30 06:09:34.749010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.748892 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-data\") pod \"0e614f11-3285-4826-aec4-ffa24b359348\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " Oct 30 06:09:34.749010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.748959 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e614f11-3285-4826-aec4-ffa24b359348-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e614f11-3285-4826-aec4-ffa24b359348-istio-podinfo\") pod \"0e614f11-3285-4826-aec4-ffa24b359348\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " Oct 30 06:09:34.749119 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749018 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e614f11-3285-4826-aec4-ffa24b359348-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e614f11-3285-4826-aec4-ffa24b359348-istiod-ca-cert\") pod \"0e614f11-3285-4826-aec4-ffa24b359348\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " Oct 30 06:09:34.749119 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749027 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-certs" pod "0e614f11-3285-4826-aec4-ffa24b359348" (UID: "0e614f11-3285-4826-aec4-ffa24b359348"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.749119 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749089 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-credential-socket" pod "0e614f11-3285-4826-aec4-ffa24b359348" (UID: "0e614f11-3285-4826-aec4-ffa24b359348"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.749300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749111 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-socket" pod "0e614f11-3285-4826-aec4-ffa24b359348" (UID: "0e614f11-3285-4826-aec4-ffa24b359348"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.749300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749167 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-data" pod "0e614f11-3285-4826-aec4-ffa24b359348" (UID: "0e614f11-3285-4826-aec4-ffa24b359348"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.749300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749203 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-envoy\") pod \"0e614f11-3285-4826-aec4-ffa24b359348\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " Oct 30 06:09:34.749300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749277 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-istio-token\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-istio-token\") pod \"0e614f11-3285-4826-aec4-ffa24b359348\" (UID: \"0e614f11-3285-4826-aec4-ffa24b359348\") " Oct 30 06:09:34.749557 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749426 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e614f11-3285-4826-aec4-ffa24b359348-istiod-ca-cert" pod "0e614f11-3285-4826-aec4-ffa24b359348" (UID: "0e614f11-3285-4826-aec4-ffa24b359348"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:34.749971 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749944 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.750104 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749976 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.750104 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.749989 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e614f11-3285-4826-aec4-ffa24b359348-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.750104 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.750003 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.750104 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.750016 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.751811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.751772 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-kube-api-access-fspmh" pod "0e614f11-3285-4826-aec4-ffa24b359348" (UID: "0e614f11-3285-4826-aec4-ffa24b359348"). InnerVolumeSpecName "kube-api-access-fspmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:34.751910 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.751864 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-envoy" pod "0e614f11-3285-4826-aec4-ffa24b359348" (UID: "0e614f11-3285-4826-aec4-ffa24b359348"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.752220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.752198 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e614f11-3285-4826-aec4-ffa24b359348-istio-podinfo" pod "0e614f11-3285-4826-aec4-ffa24b359348" (UID: "0e614f11-3285-4826-aec4-ffa24b359348"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:34.752220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.752206 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-istio-token" pod "0e614f11-3285-4826-aec4-ffa24b359348" (UID: "0e614f11-3285-4826-aec4-ffa24b359348"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:34.850983 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.850906 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e614f11-3285-4826-aec4-ffa24b359348-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.850983 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.850935 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e614f11-3285-4826-aec4-ffa24b359348-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.850983 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.850946 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.850983 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.850954 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fspmh\" (UniqueName: \"kubernetes.io/projected/0e614f11-3285-4826-aec4-ffa24b359348-kube-api-access-fspmh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.978614 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.978575 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:34.992365 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:34.992339 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:35.598841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:35.598811 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f" Oct 30 06:09:35.612085 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:35.612026 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:09:35.726077 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:35.726040 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f"] Oct 30 06:09:35.734911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:35.734879 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f28e2d-openshift-default-bdc94455c-ttz7f"] Oct 30 06:09:36.522132 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:36.522097 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e614f11-3285-4826-aec4-ffa24b359348" path="/var/lib/kubelet/pods/0e614f11-3285-4826-aec4-ffa24b359348/volumes" Oct 30 06:09:43.825230 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.825192 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx"] Oct 30 06:09:43.825999 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.825980 5072 cpu_manager.go:414] "containerMap: removing container" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" Oct 30 06:09:43.826069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.826001 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" Oct 30 06:09:43.826069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.826019 5072 cpu_manager.go:414] "containerMap: removing container" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" Oct 30 06:09:43.826069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.826024 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" Oct 30 06:09:43.826208 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.826197 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" Oct 30 06:09:43.826245 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.826213 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e614f11-3285-4826-aec4-ffa24b359348" containerName="istio-proxy" Oct 30 06:09:43.831233 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.831213 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:43.859902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.859868 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx"] Oct 30 06:09:43.861822 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.861797 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-c2be21-serving-cert\"" Oct 30 06:09:43.996393 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.996360 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea933c4-0f19-489d-a553-98d2da67114a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:43.996591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.996433 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:43.996591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.996556 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:43.996757 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:43.996731 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lldfx\" (UniqueName: \"kubernetes.io/projected/5ea933c4-0f19-489d-a553-98d2da67114a-kube-api-access-lldfx\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.098384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.098294 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lldfx\" (UniqueName: \"kubernetes.io/projected/5ea933c4-0f19-489d-a553-98d2da67114a-kube-api-access-lldfx\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.098571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.098443 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea933c4-0f19-489d-a553-98d2da67114a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.098571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.098520 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.098670 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.098613 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.098670 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:09:44.098623 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-serving-cert: secret "maas-api-e2e-shared-a-w5-c2be21-serving-cert" not found Oct 30 06:09:44.098750 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:09:44.098684 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls podName:5ea933c4-0f19-489d-a553-98d2da67114a nodeName:}" failed. No retries permitted until 2026-10-30 06:09:44.598664534 +0000 UTC m=+2344.202353770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls") pod "maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" (UID: "5ea933c4-0f19-489d-a553-98d2da67114a") : secret "maas-api-e2e-shared-a-w5-c2be21-serving-cert" not found Oct 30 06:09:44.099140 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.099106 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea933c4-0f19-489d-a553-98d2da67114a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.101598 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.101574 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.120622 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.120595 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lldfx\" (UniqueName: \"kubernetes.io/projected/5ea933c4-0f19-489d-a553-98d2da67114a-kube-api-access-lldfx\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.605480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.605386 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.608193 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.608169 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:44.755811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:44.755773 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:45.103430 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:45.103390 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx"] Oct 30 06:09:45.133901 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:09:45.133773 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea933c4_0f19_489d_a553_98d2da67114a.slice/crio-4ac993547e160e2daf6fc89594baa020eb188f6d201218ad5d47cf3b156dc226 WatchSource:0}: Error finding container 4ac993547e160e2daf6fc89594baa020eb188f6d201218ad5d47cf3b156dc226: Status 404 returned error can't find the container with id 4ac993547e160e2daf6fc89594baa020eb188f6d201218ad5d47cf3b156dc226 Oct 30 06:09:45.140690 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:45.140666 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:09:45.701504 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:45.701452 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" event={"ID":"5ea933c4-0f19-489d-a553-98d2da67114a","Type":"ContainerStarted","Data":"8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114"} Oct 30 06:09:45.701671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:45.701537 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:09:45.701671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:45.701560 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" event={"ID":"5ea933c4-0f19-489d-a553-98d2da67114a","Type":"ContainerStarted","Data":"4ac993547e160e2daf6fc89594baa020eb188f6d201218ad5d47cf3b156dc226"} Oct 30 06:09:45.744505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:45.742274 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" podStartSLOduration=2.348897618 podStartE2EDuration="2.742260072s" totalImagesPullingTime="393.362454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:45.741696682 +0000 UTC m=+2345.345385965" watchObservedRunningTime="2026-10-30 06:09:45.742260072 +0000 UTC m=+2345.345949328" Oct 30 06:09:50.229937 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:50.229907 5072 scope.go:122] "RemoveContainer" containerID="0c62c319a143899c3d2757e8b8d16071ef59d7240b40533e9c7f88cb9ca8f489" Oct 30 06:09:51.714935 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:09:51.714891 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:10:00.714114 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:10:00.714071 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:10:03.592730 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:03.592689 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5"] Oct 30 06:10:03.593116 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:03.592947 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" containerID="cri-o://425f174da38ee609a5047324bb7a4dfd7411a9e69747ae778ccf24e747f3520a" gracePeriod=30 Oct 30 06:10:08.001017 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.000978 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc"] Oct 30 06:10:08.001971 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.001939 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" containerID="cri-o://8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0" gracePeriod=30 Oct 30 06:10:08.893046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.893014 5072 generic.go:356] "Generic (PLEG): container finished" podID="9845e266-8769-4780-9b49-46c78b07aece" containerID="425f174da38ee609a5047324bb7a4dfd7411a9e69747ae778ccf24e747f3520a" exitCode=0 Oct 30 06:10:08.893224 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.893080 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" event={"ID":"9845e266-8769-4780-9b49-46c78b07aece","Type":"ContainerDied","Data":"425f174da38ee609a5047324bb7a4dfd7411a9e69747ae778ccf24e747f3520a"} Oct 30 06:10:08.919248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.919215 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:10:08.965003 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.964966 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-kube-api-access-mbxh6\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-kube-api-access-mbxh6\") pod \"9845e266-8769-4780-9b49-46c78b07aece\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " Oct 30 06:10:08.965206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.965028 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9845e266-8769-4780-9b49-46c78b07aece-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9845e266-8769-4780-9b49-46c78b07aece-istio-podinfo\") pod \"9845e266-8769-4780-9b49-46c78b07aece\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " Oct 30 06:10:08.965206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.965048 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9845e266-8769-4780-9b49-46c78b07aece-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9845e266-8769-4780-9b49-46c78b07aece-istiod-ca-cert\") pod \"9845e266-8769-4780-9b49-46c78b07aece\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " Oct 30 06:10:08.965206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.965079 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-certs\") pod \"9845e266-8769-4780-9b49-46c78b07aece\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " Oct 30 06:10:08.965206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.965139 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-envoy\") pod \"9845e266-8769-4780-9b49-46c78b07aece\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " Oct 30 06:10:08.965206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.965168 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-istio-token\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-istio-token\") pod \"9845e266-8769-4780-9b49-46c78b07aece\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " Oct 30 06:10:08.965562 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.965224 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-socket\") pod \"9845e266-8769-4780-9b49-46c78b07aece\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " Oct 30 06:10:08.965562 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.965293 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-credential-socket\") pod \"9845e266-8769-4780-9b49-46c78b07aece\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " Oct 30 06:10:08.965562 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.965333 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-data\") pod \"9845e266-8769-4780-9b49-46c78b07aece\" (UID: \"9845e266-8769-4780-9b49-46c78b07aece\") " Oct 30 06:10:08.966142 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.966115 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-data" pod "9845e266-8769-4780-9b49-46c78b07aece" (UID: "9845e266-8769-4780-9b49-46c78b07aece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:08.968715 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.968687 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-kube-api-access-mbxh6" pod "9845e266-8769-4780-9b49-46c78b07aece" (UID: "9845e266-8769-4780-9b49-46c78b07aece"). InnerVolumeSpecName "kube-api-access-mbxh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:08.970830 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.970808 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9845e266-8769-4780-9b49-46c78b07aece-istio-podinfo" pod "9845e266-8769-4780-9b49-46c78b07aece" (UID: "9845e266-8769-4780-9b49-46c78b07aece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:08.971041 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.971027 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9845e266-8769-4780-9b49-46c78b07aece-istiod-ca-cert" pod "9845e266-8769-4780-9b49-46c78b07aece" (UID: "9845e266-8769-4780-9b49-46c78b07aece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:08.971200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.971186 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-certs" pod "9845e266-8769-4780-9b49-46c78b07aece" (UID: "9845e266-8769-4780-9b49-46c78b07aece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:08.974182 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.973898 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-envoy" pod "9845e266-8769-4780-9b49-46c78b07aece" (UID: "9845e266-8769-4780-9b49-46c78b07aece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:08.976450 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.976421 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-istio-token" pod "9845e266-8769-4780-9b49-46c78b07aece" (UID: "9845e266-8769-4780-9b49-46c78b07aece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:08.977115 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.976667 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-socket" pod "9845e266-8769-4780-9b49-46c78b07aece" (UID: "9845e266-8769-4780-9b49-46c78b07aece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:08.977115 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:08.976837 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-credential-socket" pod "9845e266-8769-4780-9b49-46c78b07aece" (UID: "9845e266-8769-4780-9b49-46c78b07aece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:09.067309 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.067274 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbxh6\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-kube-api-access-mbxh6\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.067309 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.067302 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9845e266-8769-4780-9b49-46c78b07aece-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.067309 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.067312 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9845e266-8769-4780-9b49-46c78b07aece-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.067309 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.067320 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.067823 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.067329 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.067823 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.067337 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9845e266-8769-4780-9b49-46c78b07aece-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.067823 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.067344 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.067823 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.067352 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.067823 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.067360 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9845e266-8769-4780-9b49-46c78b07aece-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.902057 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.902022 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" Oct 30 06:10:09.902057 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.902049 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5" event={"ID":"9845e266-8769-4780-9b49-46c78b07aece","Type":"ContainerDied","Data":"35ed47b40ec56a5312e533f034e49417484afe92d394b74a92012060456eb5fd"} Oct 30 06:10:09.902314 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.902136 5072 scope.go:122] "RemoveContainer" containerID="425f174da38ee609a5047324bb7a4dfd7411a9e69747ae778ccf24e747f3520a" Oct 30 06:10:09.961478 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.961420 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5"] Oct 30 06:10:09.971979 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:09.971951 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-n8jx5"] Oct 30 06:10:10.520319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:10.520289 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9845e266-8769-4780-9b49-46c78b07aece" path="/var/lib/kubelet/pods/9845e266-8769-4780-9b49-46c78b07aece/volumes" Oct 30 06:10:12.789818 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:12.788820 5072 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:10:12.789818 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:12.788880 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:10:13.315964 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.315899 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:10:13.366986 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.366955 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-certs\") pod \"e4e2d554-5320-439f-a7cb-133f6c6e4780\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " Oct 30 06:10:13.367153 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367002 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-envoy\") pod \"e4e2d554-5320-439f-a7cb-133f6c6e4780\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " Oct 30 06:10:13.367153 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367051 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-token\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-token\") pod \"e4e2d554-5320-439f-a7cb-133f6c6e4780\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " Oct 30 06:10:13.367153 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367146 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-data\") pod \"e4e2d554-5320-439f-a7cb-133f6c6e4780\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " Oct 30 06:10:13.367332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367214 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-podinfo\") pod \"e4e2d554-5320-439f-a7cb-133f6c6e4780\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " Oct 30 06:10:13.367332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367249 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-certs" pod "e4e2d554-5320-439f-a7cb-133f6c6e4780" (UID: "e4e2d554-5320-439f-a7cb-133f6c6e4780"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.367332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367266 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-kube-api-access-r5mmf\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-kube-api-access-r5mmf\") pod \"e4e2d554-5320-439f-a7cb-133f6c6e4780\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " Oct 30 06:10:13.367515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367350 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-credential-socket\") pod \"e4e2d554-5320-439f-a7cb-133f6c6e4780\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " Oct 30 06:10:13.367515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367418 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-socket\") pod \"e4e2d554-5320-439f-a7cb-133f6c6e4780\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " Oct 30 06:10:13.367515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367452 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e2d554-5320-439f-a7cb-133f6c6e4780-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e2d554-5320-439f-a7cb-133f6c6e4780-istiod-ca-cert\") pod \"e4e2d554-5320-439f-a7cb-133f6c6e4780\" (UID: \"e4e2d554-5320-439f-a7cb-133f6c6e4780\") " Oct 30 06:10:13.367515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367504 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-data" pod "e4e2d554-5320-439f-a7cb-133f6c6e4780" (UID: "e4e2d554-5320-439f-a7cb-133f6c6e4780"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.367955 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.367934 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-credential-socket" pod "e4e2d554-5320-439f-a7cb-133f6c6e4780" (UID: "e4e2d554-5320-439f-a7cb-133f6c6e4780"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.368249 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.368193 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-socket" pod "e4e2d554-5320-439f-a7cb-133f6c6e4780" (UID: "e4e2d554-5320-439f-a7cb-133f6c6e4780"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.369254 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.368750 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e2d554-5320-439f-a7cb-133f6c6e4780-istiod-ca-cert" pod "e4e2d554-5320-439f-a7cb-133f6c6e4780" (UID: "e4e2d554-5320-439f-a7cb-133f6c6e4780"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:13.371082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.371061 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.371203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.371190 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.371306 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.371294 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.371400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.371388 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.371511 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.371497 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e2d554-5320-439f-a7cb-133f6c6e4780-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.377731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.377704 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-envoy" pod "e4e2d554-5320-439f-a7cb-133f6c6e4780" (UID: "e4e2d554-5320-439f-a7cb-133f6c6e4780"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.377829 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.377738 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-kube-api-access-r5mmf" pod "e4e2d554-5320-439f-a7cb-133f6c6e4780" (UID: "e4e2d554-5320-439f-a7cb-133f6c6e4780"). InnerVolumeSpecName "kube-api-access-r5mmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:13.377829 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.377770 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-token" pod "e4e2d554-5320-439f-a7cb-133f6c6e4780" (UID: "e4e2d554-5320-439f-a7cb-133f6c6e4780"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:13.379384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.379342 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-podinfo" pod "e4e2d554-5320-439f-a7cb-133f6c6e4780" (UID: "e4e2d554-5320-439f-a7cb-133f6c6e4780"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:13.473351 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.473323 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.473351 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.473348 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.473351 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.473357 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e2d554-5320-439f-a7cb-133f6c6e4780-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.473603 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.473366 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mmf\" (UniqueName: \"kubernetes.io/projected/e4e2d554-5320-439f-a7cb-133f6c6e4780-kube-api-access-r5mmf\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.938999 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.938968 5072 generic.go:356] "Generic (PLEG): container finished" podID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerID="8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0" exitCode=0 Oct 30 06:10:13.939549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.939036 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" Oct 30 06:10:13.939549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.939050 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" event={"ID":"e4e2d554-5320-439f-a7cb-133f6c6e4780","Type":"ContainerDied","Data":"8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0"} Oct 30 06:10:13.939549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.939107 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc" event={"ID":"e4e2d554-5320-439f-a7cb-133f6c6e4780","Type":"ContainerDied","Data":"6b3903863483d43e0bbba7924bf2cedd7a7de39f234a366f0a4a956e6f56ff54"} Oct 30 06:10:13.939549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.939135 5072 scope.go:122] "RemoveContainer" containerID="8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0" Oct 30 06:10:13.954611 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.954590 5072 scope.go:122] "RemoveContainer" containerID="32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96" Oct 30 06:10:13.965012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.964982 5072 scope.go:122] "RemoveContainer" containerID="8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0" Oct 30 06:10:13.965334 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:10:13.965303 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0\": container with ID starting with 8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0 not found: ID does not exist" containerID="8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0" Oct 30 06:10:13.965428 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.965345 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0"} err="failed to get container status \"8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0\": rpc error: code = NotFound desc = could not find container \"8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0\": container with ID starting with 8abc444c703f07b321740171e6f917ec1ad53c3eff12b2a7658e1965713722d0 not found: ID does not exist" Oct 30 06:10:13.965428 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.965369 5072 scope.go:122] "RemoveContainer" containerID="32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96" Oct 30 06:10:13.965777 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:10:13.965755 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96\": container with ID starting with 32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96 not found: ID does not exist" containerID="32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96" Oct 30 06:10:13.965842 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.965787 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96"} err="failed to get container status \"32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96\": rpc error: code = NotFound desc = could not find container \"32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96\": container with ID starting with 32e0df70fa4b013e6d940f98b4566942f9f37975debf2fd761b71db7022b9d96 not found: ID does not exist" Oct 30 06:10:13.987336 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.987302 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc"] Oct 30 06:10:13.998210 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:13.998175 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-kxrdc"] Oct 30 06:10:14.523165 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:14.523125 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" path="/var/lib/kubelet/pods/e4e2d554-5320-439f-a7cb-133f6c6e4780/volumes" Oct 30 06:10:23.566422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.566385 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw"] Oct 30 06:10:23.567086 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.567069 5072 cpu_manager.go:414] "containerMap: removing container" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" Oct 30 06:10:23.567086 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.567086 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" Oct 30 06:10:23.567228 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.567104 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" Oct 30 06:10:23.567228 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.567109 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" Oct 30 06:10:23.567228 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.567135 5072 cpu_manager.go:414] "containerMap: removing container" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" Oct 30 06:10:23.567228 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.567140 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" Oct 30 06:10:23.567367 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.567233 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" Oct 30 06:10:23.567367 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.567244 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9845e266-8769-4780-9b49-46c78b07aece" containerName="istio-proxy" Oct 30 06:10:23.572993 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.572967 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.593028 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.588000 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-2f1e4c-openshift-default-dockercfg-r26nk\"" Oct 30 06:10:23.604496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.604450 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw"] Oct 30 06:10:23.644596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.644536 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-credential-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.644596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.644607 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-data\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.644869 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.644754 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-token\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.644869 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.644798 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.645005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.644897 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-certs\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.645074 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.645038 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd71ff23-7006-4545-aeb3-087a3310a0bb-istiod-ca-cert\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.645149 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.645122 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlvw4\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-kube-api-access-jlvw4\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.645269 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.645248 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-podinfo\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.645367 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.645288 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-envoy\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747182 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747084 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-credential-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747182 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747133 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-data\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747231 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-token\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747262 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747336 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-certs\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747419 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd71ff23-7006-4545-aeb3-087a3310a0bb-istiod-ca-cert\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747495 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlvw4\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-kube-api-access-jlvw4\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747555 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-credential-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747566 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-data\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747590 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-podinfo\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.747618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747620 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-envoy\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.748250 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.747804 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-certs\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.748375 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.748338 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.749286 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.749203 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd71ff23-7006-4545-aeb3-087a3310a0bb-istiod-ca-cert\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.750609 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.750584 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-envoy\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.751069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.751052 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-podinfo\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.769547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.769520 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-token\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.769863 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.769842 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlvw4\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-kube-api-access-jlvw4\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:23.906811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:23.906746 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:24.270553 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:24.270528 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw"] Oct 30 06:10:24.271685 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:10:24.271658 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd71ff23_7006_4545_aeb3_087a3310a0bb.slice/crio-5e2311fa8edbdffc2c10722e129baf22757bc750a2c84faabb64f85b91f020c8 WatchSource:0}: Error finding container 5e2311fa8edbdffc2c10722e129baf22757bc750a2c84faabb64f85b91f020c8: Status 404 returned error can't find the container with id 5e2311fa8edbdffc2c10722e129baf22757bc750a2c84faabb64f85b91f020c8 Oct 30 06:10:24.274441 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:24.274398 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:10:24.274581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:24.274516 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:10:24.274581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:24.274568 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:10:25.039891 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:25.039841 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" event={"ID":"cd71ff23-7006-4545-aeb3-087a3310a0bb","Type":"ContainerStarted","Data":"8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad"} Oct 30 06:10:25.040299 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:25.039952 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" event={"ID":"cd71ff23-7006-4545-aeb3-087a3310a0bb","Type":"ContainerStarted","Data":"5e2311fa8edbdffc2c10722e129baf22757bc750a2c84faabb64f85b91f020c8"} Oct 30 06:10:25.106052 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:25.105999 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" podStartSLOduration=2.105985959 podStartE2EDuration="2.105985959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:25.092936898 +0000 UTC m=+2384.696626168" watchObservedRunningTime="2026-10-30 06:10:25.105985959 +0000 UTC m=+2384.709675217" Oct 30 06:10:25.908178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:25.908132 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:25.915269 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:25.915240 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:26.048169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:26.048123 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:26.049360 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:26.049339 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:10:29.302264 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.302234 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx"] Oct 30 06:10:29.318066 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.318030 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e2d554-5320-439f-a7cb-133f6c6e4780" containerName="istio-proxy" Oct 30 06:10:29.325048 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.325014 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:29.327374 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.327344 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx"] Oct 30 06:10:29.390787 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.390754 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-2f1e4c\"" Oct 30 06:10:29.480550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.391153 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-2f1e4c-dockercfg-c7xkt\"" Oct 30 06:10:29.496697 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.496659 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bd7978-6f64-456c-b403-108050d494e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx\" (UID: \"c0bd7978-6f64-456c-b403-108050d494e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:29.497257 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.497227 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-675lj\" (UniqueName: \"kubernetes.io/projected/c0bd7978-6f64-456c-b403-108050d494e2-kube-api-access-675lj\") pod \"payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx\" (UID: \"c0bd7978-6f64-456c-b403-108050d494e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:29.599103 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.599002 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bd7978-6f64-456c-b403-108050d494e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx\" (UID: \"c0bd7978-6f64-456c-b403-108050d494e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:29.599561 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.599533 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-675lj\" (UniqueName: \"kubernetes.io/projected/c0bd7978-6f64-456c-b403-108050d494e2-kube-api-access-675lj\") pod \"payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx\" (UID: \"c0bd7978-6f64-456c-b403-108050d494e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:29.599878 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.599856 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bd7978-6f64-456c-b403-108050d494e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx\" (UID: \"c0bd7978-6f64-456c-b403-108050d494e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:29.622170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.622138 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-675lj\" (UniqueName: \"kubernetes.io/projected/c0bd7978-6f64-456c-b403-108050d494e2-kube-api-access-675lj\") pod \"payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx\" (UID: \"c0bd7978-6f64-456c-b403-108050d494e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:29.864018 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:29.863981 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:30.205618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:30.203595 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx"] Oct 30 06:10:30.712641 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:10:30.712602 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:10:31.088302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:31.088213 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" event={"ID":"c0bd7978-6f64-456c-b403-108050d494e2","Type":"ContainerStarted","Data":"7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc"} Oct 30 06:10:31.088302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:31.088269 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:31.088302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:31.088286 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" event={"ID":"c0bd7978-6f64-456c-b403-108050d494e2","Type":"ContainerStarted","Data":"12a55344856943b6b46251b164c6929c73979082e3acedf5fe1a686793050610"} Oct 30 06:10:31.125497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:31.125418 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" podStartSLOduration=2.125400652 podStartE2EDuration="2.125400652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:31.125134867 +0000 UTC m=+2390.728824119" watchObservedRunningTime="2026-10-30 06:10:31.125400652 +0000 UTC m=+2390.729089910" Oct 30 06:10:42.153112 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.153080 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:10:42.398581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.398546 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg"] Oct 30 06:10:42.403958 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.403895 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.423423 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.422787 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-720e9c-serving-cert\"" Oct 30 06:10:42.426283 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.426238 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg"] Oct 30 06:10:42.565044 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.564958 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj4b\" (UniqueName: \"kubernetes.io/projected/d74d09be-6408-4b4f-bdbf-f36c2953566f-kube-api-access-gjj4b\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.565044 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.565016 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.565244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.565118 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.565283 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.565240 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d09be-6408-4b4f-bdbf-f36c2953566f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.677534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.673187 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.677534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.673252 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.677534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.673361 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d09be-6408-4b4f-bdbf-f36c2953566f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.677534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.673713 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj4b\" (UniqueName: \"kubernetes.io/projected/d74d09be-6408-4b4f-bdbf-f36c2953566f-kube-api-access-gjj4b\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.677534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.675377 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d09be-6408-4b4f-bdbf-f36c2953566f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.684914 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.679872 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.690603 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.690543 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.728548 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.719690 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj4b\" (UniqueName: \"kubernetes.io/projected/d74d09be-6408-4b4f-bdbf-f36c2953566f-kube-api-access-gjj4b\") pod \"maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:42.729282 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:42.729027 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:43.054522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:43.054493 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg"] Oct 30 06:10:43.056569 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:10:43.056535 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74d09be_6408_4b4f_bdbf_f36c2953566f.slice/crio-ee682e6a3adb70168247e451520ae03f69f656dd40bb90f851cdaa105d1c9883 WatchSource:0}: Error finding container ee682e6a3adb70168247e451520ae03f69f656dd40bb90f851cdaa105d1c9883: Status 404 returned error can't find the container with id ee682e6a3adb70168247e451520ae03f69f656dd40bb90f851cdaa105d1c9883 Oct 30 06:10:43.299826 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:43.299786 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" event={"ID":"d74d09be-6408-4b4f-bdbf-f36c2953566f","Type":"ContainerStarted","Data":"ee682e6a3adb70168247e451520ae03f69f656dd40bb90f851cdaa105d1c9883"} Oct 30 06:10:43.955918 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:43.955877 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp"] Oct 30 06:10:43.961535 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:43.961506 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:43.989778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:43.983453 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp"] Oct 30 06:10:43.989778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:43.984640 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-0b481b\"/\"kube-root-ca.crt\"" Oct 30 06:10:43.989778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:43.987264 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-0b481b\"/\"openshift-service-ca.crt\"" Oct 30 06:10:43.989778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:43.988668 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-0b481b\"/\"default-dockercfg-crgpc\"" Oct 30 06:10:43.989778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:43.989147 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-0b481b\"/\"auth-test-model-0b481b-kserve-self-signed-certs\"" Oct 30 06:10:44.114787 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.114741 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-home\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.114986 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.114836 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-tmp-dir\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.115296 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.115264 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-dshm\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.115361 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.115331 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-model-cache\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.115417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.115387 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb4126c-2c90-4dce-ac1b-95f90172634e-tls-certs\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.115502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.115469 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25lhp\" (UniqueName: \"kubernetes.io/projected/3eb4126c-2c90-4dce-ac1b-95f90172634e-kube-api-access-25lhp\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.217561 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.217301 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-tmp-dir\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.217561 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.217547 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-dshm\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.217811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.217600 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-model-cache\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.217811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.217630 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb4126c-2c90-4dce-ac1b-95f90172634e-tls-certs\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.217811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.217713 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25lhp\" (UniqueName: \"kubernetes.io/projected/3eb4126c-2c90-4dce-ac1b-95f90172634e-kube-api-access-25lhp\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.217987 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.217826 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-home\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.218406 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.218377 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-tmp-dir\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.218548 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.218404 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-home\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.218958 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.218933 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-model-cache\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.221528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.221500 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-dshm\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.221789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.221767 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb4126c-2c90-4dce-ac1b-95f90172634e-tls-certs\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.240675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.240646 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25lhp\" (UniqueName: \"kubernetes.io/projected/3eb4126c-2c90-4dce-ac1b-95f90172634e-kube-api-access-25lhp\") pod \"auth-test-model-0b481b-kserve-848bd74dd9-2ggpp\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.326488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.326137 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:44.351699 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.351223 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" event={"ID":"d74d09be-6408-4b4f-bdbf-f36c2953566f","Type":"ContainerStarted","Data":"141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318"} Oct 30 06:10:44.352449 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.352424 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:44.401617 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.401558 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" podStartSLOduration=2.109666099 podStartE2EDuration="2.401538609s" totalImagesPullingTime="291.87251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:44.399836648 +0000 UTC m=+2404.003525945" watchObservedRunningTime="2026-10-30 06:10:44.401538609 +0000 UTC m=+2404.005227869" Oct 30 06:10:44.734119 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:44.734081 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp"] Oct 30 06:10:44.734633 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:10:44.734606 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eb4126c_2c90_4dce_ac1b_95f90172634e.slice/crio-b1ac32a496bdb23f8d28f33225178c57019f1b39faccb54d4356b5bf38513a79 WatchSource:0}: Error finding container b1ac32a496bdb23f8d28f33225178c57019f1b39faccb54d4356b5bf38513a79: Status 404 returned error can't find the container with id b1ac32a496bdb23f8d28f33225178c57019f1b39faccb54d4356b5bf38513a79 Oct 30 06:10:45.359963 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:45.359920 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" event={"ID":"3eb4126c-2c90-4dce-ac1b-95f90172634e","Type":"ContainerStarted","Data":"370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce"} Oct 30 06:10:45.360552 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:45.359979 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" event={"ID":"3eb4126c-2c90-4dce-ac1b-95f90172634e","Type":"ContainerStarted","Data":"b1ac32a496bdb23f8d28f33225178c57019f1b39faccb54d4356b5bf38513a79"} Oct 30 06:10:45.361732 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:45.361707 5072 generic.go:356] "Generic (PLEG): container finished" podID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerID="8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d" exitCode=0 Oct 30 06:10:45.361843 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:45.361786 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" event={"ID":"b959f379-69f3-4be5-ac96-8fd4bc2211be","Type":"ContainerDied","Data":"8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d"} Oct 30 06:10:45.362282 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:45.362268 5072 scope.go:122] "RemoveContainer" containerID="8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d" Oct 30 06:10:45.363808 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:45.363782 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:10:45.363891 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:45.363847 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:10:45.363891 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:45.363881 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:10:45.398507 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:45.398444 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" podStartSLOduration=2.398425505 podStartE2EDuration="2.398425505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:45.397083536 +0000 UTC m=+2405.000772793" watchObservedRunningTime="2026-10-30 06:10:45.398425505 +0000 UTC m=+2405.002114779" Oct 30 06:10:46.347930 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:46.347894 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:46.370449 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:46.370408 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" event={"ID":"b959f379-69f3-4be5-ac96-8fd4bc2211be","Type":"ContainerStarted","Data":"c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c"} Oct 30 06:10:46.855047 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:46.855003 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:10:46.856018 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:46.855993 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:10:46.856117 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:46.856048 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 06:10:47.858552 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:47.856069 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:10:47.858552 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:47.856139 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 06:10:48.078191 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:48.078158 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:10:48.081438 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:48.081414 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:10:48.855361 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:48.855320 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:10:48.855587 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:48.855406 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 06:10:49.103270 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.103231 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774"] Oct 30 06:10:49.119840 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.119764 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.149127 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.149074 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774"] Oct 30 06:10:49.150117 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.150089 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-3abe42-openshift-default-dockercfg-6sv77\"" Oct 30 06:10:49.190970 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.190939 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:10:49.248992 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.248936 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.248992 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.248991 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.249260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.249125 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.249260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.249188 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.249379 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.249222 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.249379 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.249291 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.249379 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.249339 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30d89ef-de1f-493d-b956-70ffc9d4943c-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.249602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.249400 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.249602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.249548 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrht\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-kube-api-access-xnrht\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.351040 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.351003 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.351040 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.351042 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.351337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.351095 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.351337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.351140 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.351337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.351175 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.351337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.351212 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.351337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.351250 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30d89ef-de1f-493d-b956-70ffc9d4943c-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.351337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.351287 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.351724 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.351388 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrht\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-kube-api-access-xnrht\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.352165 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.352128 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.352165 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.352145 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.352843 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.352812 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30d89ef-de1f-493d-b956-70ffc9d4943c-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.352971 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.352935 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.353436 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.353416 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.355649 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.355622 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.355760 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.355661 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.373504 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.373405 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.373736 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.373710 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrht\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-kube-api-access-xnrht\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.464531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.464491 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:49.856143 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.856112 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:10:49.856317 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.856175 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 06:10:49.944418 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.944219 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774"] Oct 30 06:10:49.945352 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:10:49.945324 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30d89ef_de1f_493d_b956_70ffc9d4943c.slice/crio-03fdf0e379b9a4a8d662386d27cd18c9ab31a1782422efb9cdfe4d380c4d9390 WatchSource:0}: Error finding container 03fdf0e379b9a4a8d662386d27cd18c9ab31a1782422efb9cdfe4d380c4d9390: Status 404 returned error can't find the container with id 03fdf0e379b9a4a8d662386d27cd18c9ab31a1782422efb9cdfe4d380c4d9390 Oct 30 06:10:49.950446 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.949967 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:10:49.950446 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.950050 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:10:49.950446 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:49.950090 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:10:50.406333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:50.406293 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" event={"ID":"c30d89ef-de1f-493d-b956-70ffc9d4943c","Type":"ContainerStarted","Data":"0750afe02f8022dd0f7325d248463597a3df8b8eb791045466c05334639de385"} Oct 30 06:10:50.406852 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:50.406355 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" event={"ID":"c30d89ef-de1f-493d-b956-70ffc9d4943c","Type":"ContainerStarted","Data":"03fdf0e379b9a4a8d662386d27cd18c9ab31a1782422efb9cdfe4d380c4d9390"} Oct 30 06:10:50.466485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:50.465671 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:50.491601 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:50.491538 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" podStartSLOduration=1.491517452 podStartE2EDuration="1.491517452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:50.48854725 +0000 UTC m=+2410.092236509" watchObservedRunningTime="2026-10-30 06:10:50.491517452 +0000 UTC m=+2410.095206726" Oct 30 06:10:50.856262 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:50.856216 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:10:50.856527 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:50.856275 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 30 06:10:51.380823 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:51.380762 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:10:51.486683 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:51.486647 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:10:51.488901 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:51.488862 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 06:10:52.164968 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:52.164935 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:10:52.167921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:52.167895 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:10:52.465515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:52.465420 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:10:52.465675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:52.465505 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 06:10:53.465696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:53.465660 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:10:53.466167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:53.465731 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 06:10:54.348393 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:54.348357 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:54.361060 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:54.361031 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:54.465021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:54.464981 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:10:54.465194 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:54.465047 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 06:10:54.540301 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:54.540265 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:10:55.465012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:55.464979 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:10:55.465192 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:55.465029 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 06:10:56.465370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:56.465329 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:10:56.465927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:56.465410 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 06:10:57.471299 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:57.471256 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:57.471825 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:57.471730 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:10:57.472556 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:10:57.472537 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:11:00.712552 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:11:00.712515 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:11:01.817689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:01.817642 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk"] Oct 30 06:11:01.834531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:01.834488 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:01.840866 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:01.840714 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk"] Oct 30 06:11:01.849120 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:01.848582 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-2f1e4c-serving-cert\"" Oct 30 06:11:02.003103 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.003062 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04f54d9-44d0-4fae-af50-c0c6e20df84c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.003419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.003394 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.003663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.003644 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp29z\" (UniqueName: \"kubernetes.io/projected/b04f54d9-44d0-4fae-af50-c0c6e20df84c-kube-api-access-bp29z\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.003907 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.003892 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.105304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.105266 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04f54d9-44d0-4fae-af50-c0c6e20df84c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.105528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.105345 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.105528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.105410 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp29z\" (UniqueName: \"kubernetes.io/projected/b04f54d9-44d0-4fae-af50-c0c6e20df84c-kube-api-access-bp29z\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.105528 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:11:02.105513 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-serving-cert: secret "maas-api-e2e-shared-b-w5-2f1e4c-serving-cert" not found Oct 30 06:11:02.105719 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.105568 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.105719 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:11:02.105605 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls podName:b04f54d9-44d0-4fae-af50-c0c6e20df84c nodeName:}" failed. No retries permitted until 2026-10-30 06:11:02.605572004 +0000 UTC m=+2422.209261253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls") pod "maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" (UID: "b04f54d9-44d0-4fae-af50-c0c6e20df84c") : secret "maas-api-e2e-shared-b-w5-2f1e4c-serving-cert" not found Oct 30 06:11:02.106088 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.106062 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04f54d9-44d0-4fae-af50-c0c6e20df84c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.108196 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.108174 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.130283 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.130251 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp29z\" (UniqueName: \"kubernetes.io/projected/b04f54d9-44d0-4fae-af50-c0c6e20df84c-kube-api-access-bp29z\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.623145 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.615636 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.627098 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.627062 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:02.769144 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:02.769109 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:03.113515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:03.113480 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk"] Oct 30 06:11:03.114242 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:11:03.114214 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04f54d9_44d0_4fae_af50_c0c6e20df84c.slice/crio-172add3365d2b2c8e3c59ece9241aa4dbf8a6b2a081f9f4221ed9e9354893700 WatchSource:0}: Error finding container 172add3365d2b2c8e3c59ece9241aa4dbf8a6b2a081f9f4221ed9e9354893700: Status 404 returned error can't find the container with id 172add3365d2b2c8e3c59ece9241aa4dbf8a6b2a081f9f4221ed9e9354893700 Oct 30 06:11:03.573651 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:03.573605 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" event={"ID":"b04f54d9-44d0-4fae-af50-c0c6e20df84c","Type":"ContainerStarted","Data":"68347b6b4d79923bcf563bdcb09a0aa44e6bceea5e3492b834176d5f40a9499f"} Oct 30 06:11:03.573822 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:03.573676 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:03.573822 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:03.573699 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" event={"ID":"b04f54d9-44d0-4fae-af50-c0c6e20df84c","Type":"ContainerStarted","Data":"172add3365d2b2c8e3c59ece9241aa4dbf8a6b2a081f9f4221ed9e9354893700"} Oct 30 06:11:03.621702 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:03.621652 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" podStartSLOduration=2.281386326 podStartE2EDuration="2.621637841s" totalImagesPullingTime="340.251515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:03.621005494 +0000 UTC m=+2423.224694788" watchObservedRunningTime="2026-10-30 06:11:03.621637841 +0000 UTC m=+2423.225327097" Oct 30 06:11:09.587713 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:09.587684 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:11:18.058870 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.053817 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m"] Oct 30 06:11:18.060588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.060561 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.080735 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.080703 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m"] Oct 30 06:11:18.086505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.085039 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c201cd-openshift-default-dockercfg-xfx4l\"" Oct 30 06:11:18.225495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.223807 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b389edd-730e-400c-9127-8da67568ccf5-istiod-ca-cert\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.225495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.223874 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtwr\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-kube-api-access-2qtwr\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.225495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.223969 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b389edd-730e-400c-9127-8da67568ccf5-istio-podinfo\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.225495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.224004 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-istio-token\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.225495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.224034 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-certs\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.225495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.224061 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-credential-socket\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.244325 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.244284 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-envoy\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.244893 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.244866 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-data\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.245109 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.245092 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-socket\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.351602 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b389edd-730e-400c-9127-8da67568ccf5-istiod-ca-cert\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.351668 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtwr\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-kube-api-access-2qtwr\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.351753 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b389edd-730e-400c-9127-8da67568ccf5-istio-podinfo\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.351782 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-istio-token\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.351807 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-certs\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.351831 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-credential-socket\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.351941 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-envoy\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.352021 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-data\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.352085 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-socket\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.352584 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-socket\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.353209 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b389edd-730e-400c-9127-8da67568ccf5-istiod-ca-cert\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.395116 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-certs\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.395961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.395565 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-credential-socket\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.397590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.396739 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-data\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.403057 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.403028 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-envoy\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.414336 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.414293 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b389edd-730e-400c-9127-8da67568ccf5-istio-podinfo\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.416265 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.416234 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtwr\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-kube-api-access-2qtwr\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.418947 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.418922 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-istio-token\") pod \"e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:18.681421 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:18.681319 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:19.007194 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:19.007158 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m"] Oct 30 06:11:19.007910 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:11:19.007885 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b389edd_730e_400c_9127_8da67568ccf5.slice/crio-c811065776e7d7a414ea4193e590cc33073859b6ef745d9cb3046077a80deb94 WatchSource:0}: Error finding container c811065776e7d7a414ea4193e590cc33073859b6ef745d9cb3046077a80deb94: Status 404 returned error can't find the container with id c811065776e7d7a414ea4193e590cc33073859b6ef745d9cb3046077a80deb94 Oct 30 06:11:19.011021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:19.010988 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:11:19.011140 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:19.011061 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:11:19.011140 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:19.011091 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:11:19.747632 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:19.747514 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" event={"ID":"9b389edd-730e-400c-9127-8da67568ccf5","Type":"ContainerStarted","Data":"41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1"} Oct 30 06:11:19.747632 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:19.747586 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" event={"ID":"9b389edd-730e-400c-9127-8da67568ccf5","Type":"ContainerStarted","Data":"c811065776e7d7a414ea4193e590cc33073859b6ef745d9cb3046077a80deb94"} Oct 30 06:11:19.814114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:19.814048 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" podStartSLOduration=1.814034859 podStartE2EDuration="1.814034859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:19.799574458 +0000 UTC m=+2439.403263716" watchObservedRunningTime="2026-10-30 06:11:19.814034859 +0000 UTC m=+2439.417724116" Oct 30 06:11:20.682337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:20.682302 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:20.689211 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:20.689184 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:20.768090 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:20.768056 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:20.769221 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:20.769199 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:11:26.783982 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.783940 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh"] Oct 30 06:11:26.790133 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.790106 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:26.831139 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.829813 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-95498c-openshift-default-dockercfg-hvq8v\"" Oct 30 06:11:26.854666 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.854552 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh"] Oct 30 06:11:26.964488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.953861 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdj4n\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-kube-api-access-sdj4n\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:26.964488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.953931 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istiod-ca-cert\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:26.964488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.954018 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-data\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:26.964488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.954079 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-token\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:26.964488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.954155 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:26.964488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.954179 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-certs\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:26.964488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.954237 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-podinfo\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:26.964488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.954327 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-envoy\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:26.964488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:26.954358 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-credential-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.065374 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.065282 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-envoy\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.068211 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.068179 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-credential-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.068947 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.068921 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj4n\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-kube-api-access-sdj4n\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.070542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.070521 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istiod-ca-cert\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.071484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.071447 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-data\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.072057 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.071986 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-token\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.072602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.072580 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.073052 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.073034 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-certs\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.073623 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.073602 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-podinfo\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.074504 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.072977 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.074715 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.071881 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-data\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.074966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.073417 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-certs\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.075151 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.071308 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istiod-ca-cert\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.096480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.068731 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-credential-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.142576 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.142501 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-envoy\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.143353 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.143329 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-podinfo\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.165394 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.165352 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdj4n\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-kube-api-access-sdj4n\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.165686 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.165661 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-token\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.454852 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.454808 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:27.841654 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.841394 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh"] Oct 30 06:11:27.845335 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:11:27.845304 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b49a18_306a_4cbe_8fe3_1295c43ca4f3.slice/crio-3a6c87b63c8ffee926b7ee1ec3563c7b90b0c25b0edb8f27e3f8431fe01fa47f WatchSource:0}: Error finding container 3a6c87b63c8ffee926b7ee1ec3563c7b90b0c25b0edb8f27e3f8431fe01fa47f: Status 404 returned error can't find the container with id 3a6c87b63c8ffee926b7ee1ec3563c7b90b0c25b0edb8f27e3f8431fe01fa47f Oct 30 06:11:27.848412 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.848378 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:11:27.848561 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.848446 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:11:27.848561 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.848511 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:11:27.895993 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:27.895963 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" event={"ID":"60b49a18-306a-4cbe-8fe3-1295c43ca4f3","Type":"ContainerStarted","Data":"3a6c87b63c8ffee926b7ee1ec3563c7b90b0c25b0edb8f27e3f8431fe01fa47f"} Oct 30 06:11:28.907849 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:28.907814 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" event={"ID":"60b49a18-306a-4cbe-8fe3-1295c43ca4f3","Type":"ContainerStarted","Data":"1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c"} Oct 30 06:11:28.987141 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:28.987074 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" podStartSLOduration=2.98704714 podStartE2EDuration="2.98704714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:28.97541079 +0000 UTC m=+2448.579100046" watchObservedRunningTime="2026-10-30 06:11:28.98704714 +0000 UTC m=+2448.590736397" Oct 30 06:11:29.455831 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:29.455799 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:29.462415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:29.462388 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:29.950799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:29.950765 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:29.951838 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:29.951814 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:11:30.737528 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:11:30.737443 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:11:36.011201 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:36.011092 5072 generic.go:356] "Generic (PLEG): container finished" podID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerID="8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad" exitCode=0 Oct 30 06:11:36.011747 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:36.011384 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" event={"ID":"cd71ff23-7006-4545-aeb3-087a3310a0bb","Type":"ContainerDied","Data":"8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad"} Oct 30 06:11:36.012080 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:36.012056 5072 scope.go:122] "RemoveContainer" containerID="8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad" Oct 30 06:11:36.025005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:36.024963 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:11:36.025318 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:36.025280 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:11:36.025447 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:36.025353 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:11:37.023293 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:37.023256 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" event={"ID":"cd71ff23-7006-4545-aeb3-087a3310a0bb","Type":"ContainerStarted","Data":"62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e"} Oct 30 06:11:37.907807 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:37.907777 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:11:37.912625 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:37.912581 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:11:37.912903 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:37.912874 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 06:11:38.907793 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:38.907763 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:11:38.908207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:38.907824 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 06:11:39.914481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:39.908890 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:11:39.914481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:39.908963 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 06:11:40.908358 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:40.908324 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:11:40.908590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:40.908385 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 06:11:41.048792 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:41.048748 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:11:41.907833 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:41.907798 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:11:41.908062 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:41.907858 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 06:11:42.907743 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:42.907708 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:11:42.908230 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:42.907769 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 06:11:43.499580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:43.499542 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5"] Oct 30 06:11:43.913383 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:43.913350 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:11:43.914771 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:43.914748 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:11:58.534602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.534514 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" podUID="b118367c-757d-4262-a1f9-341c84dc1741" containerName="main" containerID="cri-o://729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374" gracePeriod=45 Oct 30 06:11:58.591866 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.591823 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" podUID="b118367c-757d-4262-a1f9-341c84dc1741" containerName="main" probeResult="failure" output="Get \"https://10.132.0.176:8000/ready\": dial tcp 10.132.0.176:8000: connect: connection refused" Oct 30 06:11:58.902323 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.902300 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:11:58.960531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.960488 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b118367c-757d-4262-a1f9-341c84dc1741-tls-certs\" (UniqueName: \"kubernetes.io/secret/b118367c-757d-4262-a1f9-341c84dc1741-tls-certs\") pod \"b118367c-757d-4262-a1f9-341c84dc1741\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " Oct 30 06:11:58.960729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.960571 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-model-cache\") pod \"b118367c-757d-4262-a1f9-341c84dc1741\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " Oct 30 06:11:58.960729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.960650 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-dshm\") pod \"b118367c-757d-4262-a1f9-341c84dc1741\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " Oct 30 06:11:58.960876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.960747 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-tmp-dir\") pod \"b118367c-757d-4262-a1f9-341c84dc1741\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " Oct 30 06:11:58.960939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.960889 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-model-cache" pod "b118367c-757d-4262-a1f9-341c84dc1741" (UID: "b118367c-757d-4262-a1f9-341c84dc1741"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:58.960939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.960918 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-home\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-home\") pod \"b118367c-757d-4262-a1f9-341c84dc1741\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " Oct 30 06:11:58.961054 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.960966 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b118367c-757d-4262-a1f9-341c84dc1741-kube-api-access-qnq6l\" (UniqueName: \"kubernetes.io/projected/b118367c-757d-4262-a1f9-341c84dc1741-kube-api-access-qnq6l\") pod \"b118367c-757d-4262-a1f9-341c84dc1741\" (UID: \"b118367c-757d-4262-a1f9-341c84dc1741\") " Oct 30 06:11:58.961150 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.961121 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-home" pod "b118367c-757d-4262-a1f9-341c84dc1741" (UID: "b118367c-757d-4262-a1f9-341c84dc1741"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:58.961239 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.961194 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-tmp-dir" pod "b118367c-757d-4262-a1f9-341c84dc1741" (UID: "b118367c-757d-4262-a1f9-341c84dc1741"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:58.961733 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.961699 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:11:58.961733 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.961730 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:11:58.961902 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.961770 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:11:58.963428 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.963406 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b118367c-757d-4262-a1f9-341c84dc1741-kube-api-access-qnq6l" pod "b118367c-757d-4262-a1f9-341c84dc1741" (UID: "b118367c-757d-4262-a1f9-341c84dc1741"). InnerVolumeSpecName "kube-api-access-qnq6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:58.963578 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.963488 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b118367c-757d-4262-a1f9-341c84dc1741-tls-certs" pod "b118367c-757d-4262-a1f9-341c84dc1741" (UID: "b118367c-757d-4262-a1f9-341c84dc1741"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:58.964804 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:58.964782 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-dshm" pod "b118367c-757d-4262-a1f9-341c84dc1741" (UID: "b118367c-757d-4262-a1f9-341c84dc1741"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:59.062788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.062756 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnq6l\" (UniqueName: \"kubernetes.io/projected/b118367c-757d-4262-a1f9-341c84dc1741-kube-api-access-qnq6l\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:11:59.062788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.062791 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b118367c-757d-4262-a1f9-341c84dc1741-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:11:59.062992 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.062800 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b118367c-757d-4262-a1f9-341c84dc1741-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:11:59.264788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.264694 5072 generic.go:356] "Generic (PLEG): container finished" podID="b118367c-757d-4262-a1f9-341c84dc1741" containerID="729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374" exitCode=0 Oct 30 06:11:59.264788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.264744 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" Oct 30 06:11:59.265024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.264785 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" event={"ID":"b118367c-757d-4262-a1f9-341c84dc1741","Type":"ContainerDied","Data":"729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374"} Oct 30 06:11:59.265024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.264832 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5" event={"ID":"b118367c-757d-4262-a1f9-341c84dc1741","Type":"ContainerDied","Data":"0b57adeaae3c5e803db48ddac3199c663377281e7dc084a6613b42f9be8878a5"} Oct 30 06:11:59.265024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.264855 5072 scope.go:122] "RemoveContainer" containerID="729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374" Oct 30 06:11:59.281509 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.281490 5072 scope.go:122] "RemoveContainer" containerID="729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374" Oct 30 06:11:59.281901 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:11:59.281879 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374\": container with ID starting with 729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374 not found: ID does not exist" containerID="729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374" Oct 30 06:11:59.281946 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.281909 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374"} err="failed to get container status \"729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374\": rpc error: code = NotFound desc = could not find container \"729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374\": container with ID starting with 729f88f3b925e4ec02a262523dee0df33868e69642e32ccc8403b3839661b374 not found: ID does not exist" Oct 30 06:11:59.317009 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.316978 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5"] Oct 30 06:11:59.322704 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:11:59.322678 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-3abe42/sub-test-model-3abe42-kserve-7f55689df5-8kxt5"] Oct 30 06:12:00.586584 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:00.586543 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b118367c-757d-4262-a1f9-341c84dc1741" path="/var/lib/kubelet/pods/b118367c-757d-4262-a1f9-341c84dc1741/volumes" Oct 30 06:12:00.711997 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:12:00.711955 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:12:13.313778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:13.313616 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh"] Oct 30 06:12:19.862972 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:19.862931 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm"] Oct 30 06:12:28.341335 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.341256 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" podUID="a0c5b96f-849c-421a-a843-455581f89c2b" containerName="main" containerID="cri-o://84bbc47a1e50caeed4f634f180494c9094dc43182d5d93739b98f90eba17bd1f" gracePeriod=45 Oct 30 06:12:28.453866 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.453828 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l"] Oct 30 06:12:28.454628 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.454610 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b118367c-757d-4262-a1f9-341c84dc1741" containerName="main" Oct 30 06:12:28.454689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.454629 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b118367c-757d-4262-a1f9-341c84dc1741" containerName="main" Oct 30 06:12:28.454805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.454795 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b118367c-757d-4262-a1f9-341c84dc1741" containerName="main" Oct 30 06:12:28.458450 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.458426 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.482936 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.482905 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l"] Oct 30 06:12:28.557486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.557432 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.557751 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.557724 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.557925 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.557908 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.558073 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.558057 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.558208 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.558193 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.558366 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.558350 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.558519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.558501 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.558745 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.558725 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66nhv\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-kube-api-access-66nhv\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.558866 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.558850 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.660849 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.660894 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.660945 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.660980 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661010 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661070 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661107 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661249 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66nhv\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-kube-api-access-66nhv\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661279 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661332 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661848 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661411 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661848 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661674 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661848 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661715 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.661848 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.661809 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.665721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.665698 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.666134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.666084 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.669664 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.669641 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb"] Oct 30 06:12:28.675916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.675895 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.683564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.683543 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.684130 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.684113 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66nhv\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-kube-api-access-66nhv\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.695545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.695523 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb"] Oct 30 06:12:28.716916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.716890 5072 generic.go:356] "Generic (PLEG): container finished" podID="a0c5b96f-849c-421a-a843-455581f89c2b" containerID="84bbc47a1e50caeed4f634f180494c9094dc43182d5d93739b98f90eba17bd1f" exitCode=0 Oct 30 06:12:28.717722 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.717695 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" event={"ID":"a0c5b96f-849c-421a-a843-455581f89c2b","Type":"ContainerDied","Data":"84bbc47a1e50caeed4f634f180494c9094dc43182d5d93739b98f90eba17bd1f"} Oct 30 06:12:28.717853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.717743 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" event={"ID":"a0c5b96f-849c-421a-a843-455581f89c2b","Type":"ContainerDied","Data":"6ceb6283b43954b050bbc7c5b7365bf4180d5393730cc5589c022f007b0ab0d4"} Oct 30 06:12:28.717853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.717761 5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ceb6283b43954b050bbc7c5b7365bf4180d5393730cc5589c022f007b0ab0d4" Oct 30 06:12:28.733762 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.733728 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:12:28.763004 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.762959 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-home\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-home\") pod \"a0c5b96f-849c-421a-a843-455581f89c2b\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " Oct 30 06:12:28.763273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.763246 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-home" pod "a0c5b96f-849c-421a-a843-455581f89c2b" (UID: "a0c5b96f-849c-421a-a843-455581f89c2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:28.763273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.763262 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-model-cache\") pod \"a0c5b96f-849c-421a-a843-455581f89c2b\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " Oct 30 06:12:28.763426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.763343 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c5b96f-849c-421a-a843-455581f89c2b-kube-api-access-7zwmv\" (UniqueName: \"kubernetes.io/projected/a0c5b96f-849c-421a-a843-455581f89c2b-kube-api-access-7zwmv\") pod \"a0c5b96f-849c-421a-a843-455581f89c2b\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " Oct 30 06:12:28.763426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.763403 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-dshm\") pod \"a0c5b96f-849c-421a-a843-455581f89c2b\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " Oct 30 06:12:28.763620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.763518 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-model-cache" pod "a0c5b96f-849c-421a-a843-455581f89c2b" (UID: "a0c5b96f-849c-421a-a843-455581f89c2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:28.763620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.763550 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c5b96f-849c-421a-a843-455581f89c2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c5b96f-849c-421a-a843-455581f89c2b-tls-certs\") pod \"a0c5b96f-849c-421a-a843-455581f89c2b\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " Oct 30 06:12:28.763748 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.763732 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-tmp-dir\") pod \"a0c5b96f-849c-421a-a843-455581f89c2b\" (UID: \"a0c5b96f-849c-421a-a843-455581f89c2b\") " Oct 30 06:12:28.764049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.764024 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-tmp-dir" pod "a0c5b96f-849c-421a-a843-455581f89c2b" (UID: "a0c5b96f-849c-421a-a843-455581f89c2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:28.764243 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.764217 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrkx\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-kube-api-access-mhrkx\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.764339 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.764297 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.764487 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.764544 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.764595 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.764792 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.764940 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f498f0-dbf7-4beb-85d0-22978f1255a3-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.764976 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.765027 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.765145 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.765162 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:28.765237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.765175 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:28.766523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.766499 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c5b96f-849c-421a-a843-455581f89c2b-tls-certs" pod "a0c5b96f-849c-421a-a843-455581f89c2b" (UID: "a0c5b96f-849c-421a-a843-455581f89c2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:28.766732 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.766716 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-dshm" pod "a0c5b96f-849c-421a-a843-455581f89c2b" (UID: "a0c5b96f-849c-421a-a843-455581f89c2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:28.766825 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.766812 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c5b96f-849c-421a-a843-455581f89c2b-kube-api-access-7zwmv" pod "a0c5b96f-849c-421a-a843-455581f89c2b" (UID: "a0c5b96f-849c-421a-a843-455581f89c2b"). InnerVolumeSpecName "kube-api-access-7zwmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:28.776065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.776038 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:28.866921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.866879 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f498f0-dbf7-4beb-85d0-22978f1255a3-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.866921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.866920 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.867124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.866956 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.867124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867001 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrkx\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-kube-api-access-mhrkx\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.867124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867041 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.867124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867074 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.867124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867097 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.867124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867125 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.867749 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867275 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.867749 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867380 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zwmv\" (UniqueName: \"kubernetes.io/projected/a0c5b96f-849c-421a-a843-455581f89c2b-kube-api-access-7zwmv\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:28.867749 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867392 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0c5b96f-849c-421a-a843-455581f89c2b-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:28.867749 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867414 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0c5b96f-849c-421a-a843-455581f89c2b-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:28.868042 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.867991 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.868726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.868175 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.868726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.868407 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.868726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.868424 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f498f0-dbf7-4beb-85d0-22978f1255a3-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.868726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.868677 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.872173 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.871375 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.872173 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.871731 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.889834 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.889801 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:28.890733 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:28.890711 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrkx\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-kube-api-access-mhrkx\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:29.054818 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.054733 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:29.135932 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.135905 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l"] Oct 30 06:12:29.138567 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:12:29.138533 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bbf04c_8a20_4c7e_a61e_94e9db4b2ed3.slice/crio-bc71d3a6835ad5b97dab6c0a3db9bf386f7fa9af6d42fa5164f0c9d6db940c41 WatchSource:0}: Error finding container bc71d3a6835ad5b97dab6c0a3db9bf386f7fa9af6d42fa5164f0c9d6db940c41: Status 404 returned error can't find the container with id bc71d3a6835ad5b97dab6c0a3db9bf386f7fa9af6d42fa5164f0c9d6db940c41 Oct 30 06:12:29.143012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.142966 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:29.143128 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.143064 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:29.143128 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.143109 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:29.439484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.432638 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:29.439484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.432763 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:29.439484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.432808 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:29.439484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.439262 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb"] Oct 30 06:12:29.727123 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.727028 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" event={"ID":"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3","Type":"ContainerStarted","Data":"60522e0d0c6e8b19e23b165632614349fe8b7a09a9e75c96db9388a007032c81"} Oct 30 06:12:29.727394 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.727356 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" event={"ID":"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3","Type":"ContainerStarted","Data":"bc71d3a6835ad5b97dab6c0a3db9bf386f7fa9af6d42fa5164f0c9d6db940c41"} Oct 30 06:12:29.737370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.731004 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh" Oct 30 06:12:29.737370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.731060 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" event={"ID":"43f498f0-dbf7-4beb-85d0-22978f1255a3","Type":"ContainerStarted","Data":"87d078dd3a579c1fac2e93b53a5bdbbb5218b6d982beecf02c3f72eec7512206"} Oct 30 06:12:29.737370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.731146 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" event={"ID":"43f498f0-dbf7-4beb-85d0-22978f1255a3","Type":"ContainerStarted","Data":"0ddd710897437cf4fb26e8f263c578f0cf5f0346d11001d751dbf8dd3253a889"} Oct 30 06:12:29.772340 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.772289 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" podStartSLOduration=1.77227602 podStartE2EDuration="1.77227602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:29.769582812 +0000 UTC m=+2509.373272070" watchObservedRunningTime="2026-10-30 06:12:29.77227602 +0000 UTC m=+2509.375965276" Oct 30 06:12:29.776147 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.776117 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:29.816021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.815960 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" podStartSLOduration=1.815942321 podStartE2EDuration="1.815942321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:29.810442661 +0000 UTC m=+2509.414131919" watchObservedRunningTime="2026-10-30 06:12:29.815942321 +0000 UTC m=+2509.419631579" Oct 30 06:12:29.847238 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.847208 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh"] Oct 30 06:12:29.850887 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:29.850861 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b8f87/auth-test-model-6b8f87-kserve-57fc875657-6pgdh"] Oct 30 06:12:30.055568 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:30.055433 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:30.056380 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:30.056352 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:12:30.056617 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:30.056407 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 06:12:30.581905 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:30.581874 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c5b96f-849c-421a-a843-455581f89c2b" path="/var/lib/kubelet/pods/a0c5b96f-849c-421a-a843-455581f89c2b/volumes" Oct 30 06:12:30.781673 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:30.780859 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:12:30.781673 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:30.780916 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 06:12:30.783196 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:12:30.783162 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:12:31.062492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:31.061900 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:12:31.062492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:31.061982 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 06:12:31.778482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:31.777533 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:12:31.778482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:31.777605 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 06:12:32.061538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:32.060577 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:12:32.061538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:32.060660 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 06:12:32.776590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:32.776548 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:12:32.776794 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:32.776625 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 06:12:33.063544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:33.062568 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:12:33.063544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:33.062628 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 06:12:33.779372 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:33.778842 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:12:33.779372 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:33.778919 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 06:12:34.058219 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:34.058122 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:12:34.058525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:34.058491 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 06:12:34.776944 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:34.776900 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:12:34.777496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:34.776951 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 06:12:34.895788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:34.895747 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" podUID="f47e16b1-6b33-43bf-804f-865ab51826df" containerName="main" containerID="cri-o://087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587" gracePeriod=45 Oct 30 06:12:35.057630 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.057130 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:12:35.057630 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.057193 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 06:12:35.130484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.129845 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj"] Oct 30 06:12:35.144782 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.141170 5072 cpu_manager.go:414] "containerMap: removing container" podUID="a0c5b96f-849c-421a-a843-455581f89c2b" containerName="main" Oct 30 06:12:35.144782 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.141206 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c5b96f-849c-421a-a843-455581f89c2b" containerName="main" Oct 30 06:12:35.144782 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.141412 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c5b96f-849c-421a-a843-455581f89c2b" containerName="main" Oct 30 06:12:35.154167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.154140 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.186495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.173012 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-b4ca14-openshift-default-dockercfg-fjppn\"" Oct 30 06:12:35.201526 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.200741 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj"] Oct 30 06:12:35.305505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.304539 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.305505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.304610 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-envoy\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.305505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.304733 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-token\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.305505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.304767 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istiod-ca-cert\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.305505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.304865 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpzq\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-kube-api-access-jvpzq\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.305505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.304894 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-credential-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.305505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.304919 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-data\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.305505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.304959 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-certs\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.305505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.305041 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-podinfo\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.408346 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-podinfo\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.412549 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.412619 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-envoy\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.412762 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-token\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.412798 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istiod-ca-cert\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.412909 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpzq\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-kube-api-access-jvpzq\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.412943 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-credential-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.412968 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-data\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.413014 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-certs\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.413562 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-certs\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.408510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.413826 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.450491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.448921 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istiod-ca-cert\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.450491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.449184 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-credential-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.450491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.449416 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-data\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.478326 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.477157 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-envoy\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.478326 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.478069 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-podinfo\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.485853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.485794 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpzq\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-kube-api-access-jvpzq\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.486206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.486186 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-token\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.501271 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.497152 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:35.661045 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.660940 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:12:35.753525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.745007 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-tmp-dir\") pod \"f47e16b1-6b33-43bf-804f-865ab51826df\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " Oct 30 06:12:35.753525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.745091 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47e16b1-6b33-43bf-804f-865ab51826df-kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/f47e16b1-6b33-43bf-804f-865ab51826df-kube-api-access-s5wmz\") pod \"f47e16b1-6b33-43bf-804f-865ab51826df\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " Oct 30 06:12:35.753525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.745147 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-dshm\") pod \"f47e16b1-6b33-43bf-804f-865ab51826df\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " Oct 30 06:12:35.753525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.745240 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f47e16b1-6b33-43bf-804f-865ab51826df-tls-certs\" (UniqueName: \"kubernetes.io/secret/f47e16b1-6b33-43bf-804f-865ab51826df-tls-certs\") pod \"f47e16b1-6b33-43bf-804f-865ab51826df\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " Oct 30 06:12:35.753525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.745487 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-home\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-home\") pod \"f47e16b1-6b33-43bf-804f-865ab51826df\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " Oct 30 06:12:35.753525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.745560 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-model-cache\") pod \"f47e16b1-6b33-43bf-804f-865ab51826df\" (UID: \"f47e16b1-6b33-43bf-804f-865ab51826df\") " Oct 30 06:12:35.753525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.748528 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-model-cache" pod "f47e16b1-6b33-43bf-804f-865ab51826df" (UID: "f47e16b1-6b33-43bf-804f-865ab51826df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:35.755791 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.754505 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-home" pod "f47e16b1-6b33-43bf-804f-865ab51826df" (UID: "f47e16b1-6b33-43bf-804f-865ab51826df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:35.755791 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.754717 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-tmp-dir" pod "f47e16b1-6b33-43bf-804f-865ab51826df" (UID: "f47e16b1-6b33-43bf-804f-865ab51826df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:35.755791 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.755534 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-dshm" pod "f47e16b1-6b33-43bf-804f-865ab51826df" (UID: "f47e16b1-6b33-43bf-804f-865ab51826df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:35.764003 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.763372 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f47e16b1-6b33-43bf-804f-865ab51826df-tls-certs" pod "f47e16b1-6b33-43bf-804f-865ab51826df" (UID: "f47e16b1-6b33-43bf-804f-865ab51826df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:35.776674 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.776642 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:12:35.776811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.776697 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 06:12:35.782820 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.781998 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47e16b1-6b33-43bf-804f-865ab51826df-kube-api-access-s5wmz" pod "f47e16b1-6b33-43bf-804f-865ab51826df" (UID: "f47e16b1-6b33-43bf-804f-865ab51826df"). InnerVolumeSpecName "kube-api-access-s5wmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:35.853169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.849006 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:35.853169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.849038 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5wmz\" (UniqueName: \"kubernetes.io/projected/f47e16b1-6b33-43bf-804f-865ab51826df-kube-api-access-s5wmz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:35.853169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.849053 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:35.853169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.849068 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47e16b1-6b33-43bf-804f-865ab51826df-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:35.853169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.849082 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:35.853169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.849096 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47e16b1-6b33-43bf-804f-865ab51826df-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:12:35.859330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.859294 5072 generic.go:356] "Generic (PLEG): container finished" podID="f47e16b1-6b33-43bf-804f-865ab51826df" containerID="087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587" exitCode=0 Oct 30 06:12:35.859529 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.859388 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" event={"ID":"f47e16b1-6b33-43bf-804f-865ab51826df","Type":"ContainerDied","Data":"087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587"} Oct 30 06:12:35.859529 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.859425 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" event={"ID":"f47e16b1-6b33-43bf-804f-865ab51826df","Type":"ContainerDied","Data":"a9f15664a0ee6f469de1367cccc417f6fbd005309a5ba252dece696a2ddc1f46"} Oct 30 06:12:35.859529 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.859448 5072 scope.go:122] "RemoveContainer" containerID="087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587" Oct 30 06:12:35.859758 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.859630 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" Oct 30 06:12:35.892662 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.892631 5072 scope.go:122] "RemoveContainer" containerID="087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587" Oct 30 06:12:35.893305 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:12:35.893267 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587\": container with ID starting with 087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587 not found: ID does not exist" containerID="087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587" Oct 30 06:12:35.893426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.893316 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587"} err="failed to get container status \"087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587\": rpc error: code = NotFound desc = could not find container \"087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587\": container with ID starting with 087ebda2bc46c5b7ad4062fd69d731229a303c7e49df7f4941b1de7040c0e587 not found: ID does not exist" Oct 30 06:12:35.933133 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.933047 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm"] Oct 30 06:12:35.936700 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.936668 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm"] Oct 30 06:12:35.958534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.955071 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj"] Oct 30 06:12:35.958534 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:12:35.956799 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784081e5_9fbc_4a1a_9763_1fd9d9cf33a0.slice/crio-dd561decb53f58bc1113b70ef350fbd5b936343847c3e24867ce25e593786794 WatchSource:0}: Error finding container dd561decb53f58bc1113b70ef350fbd5b936343847c3e24867ce25e593786794: Status 404 returned error can't find the container with id dd561decb53f58bc1113b70ef350fbd5b936343847c3e24867ce25e593786794 Oct 30 06:12:35.962695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.960371 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:35.962695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.960432 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:35.962695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:35.960482 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:36.055807 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.055764 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:12:36.056090 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.056061 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 06:12:36.346283 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.346188 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f"] Oct 30 06:12:36.350535 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.347034 5072 cpu_manager.go:414] "containerMap: removing container" podUID="f47e16b1-6b33-43bf-804f-865ab51826df" containerName="main" Oct 30 06:12:36.350535 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.347051 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47e16b1-6b33-43bf-804f-865ab51826df" containerName="main" Oct 30 06:12:36.350535 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.347169 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47e16b1-6b33-43bf-804f-865ab51826df" containerName="main" Oct 30 06:12:36.351101 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.351079 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.378675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.372809 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-02ceed-openshift-default-dockercfg-m7z89\"" Oct 30 06:12:36.378675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.372878 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f"] Oct 30 06:12:36.461196 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.461155 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11307d4e-323b-4246-9903-c9eab92335a7-istio-podinfo\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.461407 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.461239 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jvnx\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-kube-api-access-4jvnx\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.461407 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.461386 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.461581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.461422 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-credential-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.461581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.461452 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-data\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.461581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.461523 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-certs\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.461816 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.461591 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-envoy\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.461816 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.461616 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-istio-token\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.461816 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.461646 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11307d4e-323b-4246-9903-c9eab92335a7-istiod-ca-cert\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.524642 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.524611 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47e16b1-6b33-43bf-804f-865ab51826df" path="/var/lib/kubelet/pods/f47e16b1-6b33-43bf-804f-865ab51826df/volumes" Oct 30 06:12:36.563432 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.563390 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11307d4e-323b-4246-9903-c9eab92335a7-istio-podinfo\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.563712 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.563552 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jvnx\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-kube-api-access-4jvnx\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.563792 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.563745 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.564093 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.563804 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-credential-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.564093 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.563841 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-data\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.564093 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.563913 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-certs\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.564093 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.564012 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-envoy\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.564093 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.564057 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-istio-token\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.564093 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.564090 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11307d4e-323b-4246-9903-c9eab92335a7-istiod-ca-cert\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.565317 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.565272 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-data\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.565851 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.565822 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11307d4e-323b-4246-9903-c9eab92335a7-istiod-ca-cert\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.567214 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.567187 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.569096 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.569072 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-certs\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.569774 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.569597 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-credential-socket\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.571919 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.571892 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11307d4e-323b-4246-9903-c9eab92335a7-istio-podinfo\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.573789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.573764 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-envoy\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.591516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.588836 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-istio-token\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.591516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.589191 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jvnx\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-kube-api-access-4jvnx\") pod \"e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.599616 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.599530 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-02ceed/rate-test-model-02ceed-kserve-79869fb589-zsptm" podUID="f47e16b1-6b33-43bf-804f-865ab51826df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.180:8000/ready\": context deadline exceeded" Oct 30 06:12:36.694484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.682904 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:36.785483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.784527 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:36.785483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.785249 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:36.790503 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.786230 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:12:36.877470 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.877427 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" event={"ID":"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0","Type":"ContainerStarted","Data":"69496c53bbe91993a6b71a0753ca0a5b3c63b621445fdbf75cba475aec03eb5d"} Oct 30 06:12:36.877670 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.877507 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" event={"ID":"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0","Type":"ContainerStarted","Data":"dd561decb53f58bc1113b70ef350fbd5b936343847c3e24867ce25e593786794"} Oct 30 06:12:36.975839 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:36.975741 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" podStartSLOduration=1.975720254 podStartE2EDuration="1.975720254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:36.970552747 +0000 UTC m=+2516.574242006" watchObservedRunningTime="2026-10-30 06:12:36.975720254 +0000 UTC m=+2516.579409513" Oct 30 06:12:37.070005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.069972 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:37.071500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.071438 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:37.074925 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.074891 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f"] Oct 30 06:12:37.076969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.076927 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:37.077078 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.077042 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:37.077154 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.077092 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:12:37.077909 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.077885 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:12:37.078496 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:12:37.078446 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11307d4e_323b_4246_9903_c9eab92335a7.slice/crio-cf543e4af74c6b30e6382275a46f6576e8e1d5242e8064bae4da1cfa790f0dfe WatchSource:0}: Error finding container cf543e4af74c6b30e6382275a46f6576e8e1d5242e8064bae4da1cfa790f0dfe: Status 404 returned error can't find the container with id cf543e4af74c6b30e6382275a46f6576e8e1d5242e8064bae4da1cfa790f0dfe Oct 30 06:12:37.501501 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.501433 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:37.503219 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.503186 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:12:37.503360 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.503263 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 06:12:37.889758 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.889720 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" event={"ID":"11307d4e-323b-4246-9903-c9eab92335a7","Type":"ContainerStarted","Data":"2b8a945c1d056cfdeb5b80c265c316ffed2c526bdc9a72f23e6ef5e0526440d4"} Oct 30 06:12:37.890231 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.889785 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" event={"ID":"11307d4e-323b-4246-9903-c9eab92335a7","Type":"ContainerStarted","Data":"cf543e4af74c6b30e6382275a46f6576e8e1d5242e8064bae4da1cfa790f0dfe"} Oct 30 06:12:37.935049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:37.934911 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" podStartSLOduration=1.934892771 podStartE2EDuration="1.934892771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:37.932961789 +0000 UTC m=+2517.536651058" watchObservedRunningTime="2026-10-30 06:12:37.934892771 +0000 UTC m=+2517.538582031" Oct 30 06:12:38.501754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:38.501719 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:12:38.501945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:38.501771 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 06:12:38.683906 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:38.683873 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:38.685555 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:38.685527 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:12:38.685634 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:38.685584 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:12:39.505490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:39.501483 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:12:39.505490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:39.501563 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 06:12:39.684148 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:39.684108 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:12:39.684333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:39.684160 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:12:40.501385 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:40.501351 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:12:40.501698 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:40.501665 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 06:12:40.683899 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:40.683867 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:12:40.684297 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:40.683926 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:12:41.502802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:41.502772 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:12:41.503043 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:41.503021 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 06:12:41.684152 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:41.684122 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:12:41.684659 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:41.684631 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:12:42.501631 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:42.501594 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:12:42.501816 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:42.501663 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 06:12:42.683863 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:42.683823 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:12:42.684024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:42.683891 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:12:43.526485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:43.523312 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:43.526485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:43.524940 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:43.550846 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:43.550785 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:12:43.683775 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:43.683738 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:12:43.683987 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:43.683818 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 06:12:44.688499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:44.688447 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:44.688984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:44.688918 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:12:44.689619 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:12:44.689599 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:13:00.726192 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:13:00.726158 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:13:04.506585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:04.506538 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" podUID="3eb4126c-2c90-4dce-ac1b-95f90172634e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": context deadline exceeded" Oct 30 06:13:04.646487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:04.637520 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:04.707524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:04.706842 5072 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:04.883564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:04.878979 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:04.962524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:04.944590 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:14.953228 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:14.953194 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:13:14.953816 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:14.953247 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 06:13:15.484239 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.484199 5072 generic.go:356] "Generic (PLEG): container finished" podID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerID="1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c" exitCode=0 Oct 30 06:13:15.484446 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.484259 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" event={"ID":"60b49a18-306a-4cbe-8fe3-1295c43ca4f3","Type":"ContainerDied","Data":"1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c"} Oct 30 06:13:15.484864 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.484841 5072 scope.go:122] "RemoveContainer" containerID="1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c" Oct 30 06:13:15.486305 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.486287 5072 generic.go:356] "Generic (PLEG): container finished" podID="9b389edd-730e-400c-9127-8da67568ccf5" containerID="41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1" exitCode=0 Oct 30 06:13:15.486399 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.486323 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" event={"ID":"9b389edd-730e-400c-9127-8da67568ccf5","Type":"ContainerDied","Data":"41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1"} Oct 30 06:13:15.486693 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.486679 5072 scope.go:122] "RemoveContainer" containerID="41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1" Oct 30 06:13:15.486887 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.486863 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:13:15.486957 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.486941 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:13:15.487010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.486995 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:13:15.488434 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.488411 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:13:15.488525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.488510 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:13:15.488566 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:15.488551 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:13:16.496012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:16.495971 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" event={"ID":"9b389edd-730e-400c-9127-8da67568ccf5","Type":"ContainerStarted","Data":"1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d"} Oct 30 06:13:16.498526 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:16.498499 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" event={"ID":"60b49a18-306a-4cbe-8fe3-1295c43ca4f3","Type":"ContainerStarted","Data":"001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f"} Oct 30 06:13:16.684024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:16.683823 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:13:16.684803 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:16.684755 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:13:16.684979 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:16.684809 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 06:13:17.456486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:17.455885 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:13:17.456486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:17.456010 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:13:17.456486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:17.456054 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 06:13:17.682368 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:17.682319 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:13:17.682368 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:17.682369 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 06:13:18.459396 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:18.458940 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:13:18.459396 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:18.459003 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 06:13:18.682373 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:18.682332 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:13:18.682905 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:18.682390 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 06:13:19.455301 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.455260 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:13:19.455642 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.455606 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 06:13:19.679364 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.678310 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n"] Oct 30 06:13:19.681774 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.681743 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:13:19.681983 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.681958 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 06:13:19.685906 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.685885 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.761542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.758145 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-0b481b-openshift-default-dockercfg-nrg7r\"" Oct 30 06:13:19.782549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.782511 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n"] Oct 30 06:13:19.840484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.836546 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.840484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.836609 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjsc\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-kube-api-access-zrjsc\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.877211 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.877171 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.877538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.877513 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.877715 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.877696 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.877827 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.877802 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.877901 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.877858 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.878377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.878182 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af48b8-8385-45af-b091-f1788b83604e-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.878377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.878248 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af48b8-8385-45af-b091-f1788b83604e-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.984689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.984106 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af48b8-8385-45af-b091-f1788b83604e-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.984689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.984183 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af48b8-8385-45af-b091-f1788b83604e-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.984689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.984236 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.984689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.984268 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjsc\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-kube-api-access-zrjsc\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.984689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.984342 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.984689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.984366 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.984689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.984409 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.984689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.984441 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.984689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.984492 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.985334 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.985262 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af48b8-8385-45af-b091-f1788b83604e-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.989396 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.985982 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.989396 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.986132 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.989396 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.986508 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.989396 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.986639 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.990605 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.989805 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:19.990605 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:19.989885 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af48b8-8385-45af-b091-f1788b83604e-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:20.006667 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.006628 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjsc\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-kube-api-access-zrjsc\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:20.007088 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.007047 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:20.017981 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.017733 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:20.589486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.579384 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:13:20.589753 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.589603 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 06:13:20.693942 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.693911 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:13:20.694491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.694439 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 06:13:20.697044 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.697016 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n"] Oct 30 06:13:20.697516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.697488 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:13:20.697610 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.697543 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:13:20.697610 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.697571 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:13:20.772489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:20.771441 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:13:21.458671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:21.458438 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:13:21.458671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:21.458519 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 06:13:21.604932 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:21.604887 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" event={"ID":"60af48b8-8385-45af-b091-f1788b83604e","Type":"ContainerStarted","Data":"b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810"} Oct 30 06:13:21.605156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:21.604968 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" event={"ID":"60af48b8-8385-45af-b091-f1788b83604e","Type":"ContainerStarted","Data":"8d14a4a35006d117e5de4dce6d809d631cf903aa188393bcac7de61f44cc96f8"} Oct 30 06:13:21.682249 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:21.682209 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:13:21.682449 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:21.682280 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 06:13:21.693212 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:21.693166 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podStartSLOduration=2.693152412 podStartE2EDuration="2.693152412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:21.678884634 +0000 UTC m=+2561.282573891" watchObservedRunningTime="2026-10-30 06:13:21.693152412 +0000 UTC m=+2561.296841666" Oct 30 06:13:22.020215 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:22.020178 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:22.022507 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:22.022472 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 06:13:22.022654 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:22.022533 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 06:13:22.460869 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:22.460835 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:13:22.461899 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:22.461829 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:13:22.462974 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:22.462952 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:13:22.696336 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:22.696304 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:13:22.697706 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:22.697682 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:13:23.023483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:23.022839 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 06:13:23.023483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:23.022900 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 06:13:24.039486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:24.036715 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 06:13:24.039486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:24.036774 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 06:13:25.032484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:25.031863 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 06:13:25.032484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:25.031929 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 06:13:26.034895 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:26.034858 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 06:13:26.035549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:26.035514 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 06:13:27.031486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:27.028937 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 06:13:27.031486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:27.028997 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 06:13:28.019762 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:28.019727 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 30 06:13:28.020273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:28.019787 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 30 06:13:29.026884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:29.026848 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:29.027405 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:29.027360 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:29.028181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:29.028151 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:13:30.710823 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:13:30.710783 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:13:31.599397 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.580599 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z"] Oct 30 06:13:31.618988 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.618955 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z"] Oct 30 06:13:31.620708 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.620677 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:13:31.651374 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.651345 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c201cd-dockercfg-vmn8x\"" Oct 30 06:13:31.652390 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.651546 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c201cd\"" Oct 30 06:13:31.712499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.710748 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d8477b-0754-4ff5-9aac-eab6828775ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z\" (UID: \"d5d8477b-0754-4ff5-9aac-eab6828775ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:13:31.712499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.710812 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmwk\" (UniqueName: \"kubernetes.io/projected/d5d8477b-0754-4ff5-9aac-eab6828775ee-kube-api-access-gmmwk\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z\" (UID: \"d5d8477b-0754-4ff5-9aac-eab6828775ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:13:31.812311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.812267 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d8477b-0754-4ff5-9aac-eab6828775ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z\" (UID: \"d5d8477b-0754-4ff5-9aac-eab6828775ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:13:31.812680 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.812648 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmwk\" (UniqueName: \"kubernetes.io/projected/d5d8477b-0754-4ff5-9aac-eab6828775ee-kube-api-access-gmmwk\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z\" (UID: \"d5d8477b-0754-4ff5-9aac-eab6828775ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:13:31.813346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.813319 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d8477b-0754-4ff5-9aac-eab6828775ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z\" (UID: \"d5d8477b-0754-4ff5-9aac-eab6828775ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:13:31.834990 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.834964 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmwk\" (UniqueName: \"kubernetes.io/projected/d5d8477b-0754-4ff5-9aac-eab6828775ee-kube-api-access-gmmwk\") pod \"payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z\" (UID: \"d5d8477b-0754-4ff5-9aac-eab6828775ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:13:31.965143 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:31.965113 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:13:32.337686 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:32.337656 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z"] Oct 30 06:13:32.339383 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:13:32.339357 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d8477b_0754_4ff5_9aac_eab6828775ee.slice/crio-0cda668fa9901f546d9bf7dc8eff0335d6fa7d939208c9c9ff0ab891fe3aa5ce WatchSource:0}: Error finding container 0cda668fa9901f546d9bf7dc8eff0335d6fa7d939208c9c9ff0ab891fe3aa5ce: Status 404 returned error can't find the container with id 0cda668fa9901f546d9bf7dc8eff0335d6fa7d939208c9c9ff0ab891fe3aa5ce Oct 30 06:13:32.756449 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:32.756401 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" event={"ID":"d5d8477b-0754-4ff5-9aac-eab6828775ee","Type":"ContainerStarted","Data":"0cda668fa9901f546d9bf7dc8eff0335d6fa7d939208c9c9ff0ab891fe3aa5ce"} Oct 30 06:13:34.781943 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:34.781903 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" event={"ID":"d5d8477b-0754-4ff5-9aac-eab6828775ee","Type":"ContainerStarted","Data":"3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6"} Oct 30 06:13:34.782451 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:34.781988 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:13:34.863883 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:34.863822 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" podStartSLOduration=2.184312283 podStartE2EDuration="3.863805064s" totalImagesPullingTime="1.679492781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:34.862502333 +0000 UTC m=+2574.466191584" watchObservedRunningTime="2026-10-30 06:13:34.863805064 +0000 UTC m=+2574.467494321" Oct 30 06:13:45.792057 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:13:45.792025 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:14:00.713989 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:14:00.713942 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:14:10.974793 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:10.974754 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp"] Oct 30 06:14:26.018641 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.018585 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" podUID="3eb4126c-2c90-4dce-ac1b-95f90172634e" containerName="main" containerID="cri-o://370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce" gracePeriod=45 Oct 30 06:14:26.550663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.550634 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:14:26.583610 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.583570 5072 generic.go:356] "Generic (PLEG): container finished" podID="3eb4126c-2c90-4dce-ac1b-95f90172634e" containerID="370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce" exitCode=0 Oct 30 06:14:26.583827 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.583804 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" event={"ID":"3eb4126c-2c90-4dce-ac1b-95f90172634e","Type":"ContainerDied","Data":"370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce"} Oct 30 06:14:26.583934 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.583851 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" event={"ID":"3eb4126c-2c90-4dce-ac1b-95f90172634e","Type":"ContainerDied","Data":"b1ac32a496bdb23f8d28f33225178c57019f1b39faccb54d4356b5bf38513a79"} Oct 30 06:14:26.583934 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.583876 5072 scope.go:122] "RemoveContainer" containerID="370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce" Oct 30 06:14:26.584081 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.584043 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" Oct 30 06:14:26.604124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.604089 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb4126c-2c90-4dce-ac1b-95f90172634e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb4126c-2c90-4dce-ac1b-95f90172634e-tls-certs\") pod \"3eb4126c-2c90-4dce-ac1b-95f90172634e\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " Oct 30 06:14:26.604393 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.604372 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-dshm\") pod \"3eb4126c-2c90-4dce-ac1b-95f90172634e\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " Oct 30 06:14:26.605203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.604630 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-tmp-dir\") pod \"3eb4126c-2c90-4dce-ac1b-95f90172634e\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " Oct 30 06:14:26.685528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.685496 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb4126c-2c90-4dce-ac1b-95f90172634e-kube-api-access-25lhp\" (UniqueName: \"kubernetes.io/projected/3eb4126c-2c90-4dce-ac1b-95f90172634e-kube-api-access-25lhp\") pod \"3eb4126c-2c90-4dce-ac1b-95f90172634e\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " Oct 30 06:14:26.685839 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.685814 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-model-cache\") pod \"3eb4126c-2c90-4dce-ac1b-95f90172634e\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " Oct 30 06:14:26.686109 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.686091 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-home\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-home\") pod \"3eb4126c-2c90-4dce-ac1b-95f90172634e\" (UID: \"3eb4126c-2c90-4dce-ac1b-95f90172634e\") " Oct 30 06:14:26.704484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.682791 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-tmp-dir" pod "3eb4126c-2c90-4dce-ac1b-95f90172634e" (UID: "3eb4126c-2c90-4dce-ac1b-95f90172634e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:26.704739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.689163 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-home" pod "3eb4126c-2c90-4dce-ac1b-95f90172634e" (UID: "3eb4126c-2c90-4dce-ac1b-95f90172634e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:26.713479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.627661 5072 scope.go:122] "RemoveContainer" containerID="370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce" Oct 30 06:14:26.713681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.713610 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-model-cache" pod "3eb4126c-2c90-4dce-ac1b-95f90172634e" (UID: "3eb4126c-2c90-4dce-ac1b-95f90172634e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:26.714658 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.714610 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb4126c-2c90-4dce-ac1b-95f90172634e-tls-certs" pod "3eb4126c-2c90-4dce-ac1b-95f90172634e" (UID: "3eb4126c-2c90-4dce-ac1b-95f90172634e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:26.755486 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:14:26.749756 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce\": container with ID starting with 370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce not found: ID does not exist" containerID="370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce" Oct 30 06:14:26.755486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.749807 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce"} err="failed to get container status \"370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce\": rpc error: code = NotFound desc = could not find container \"370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce\": container with ID starting with 370745fdf65921714621f7f01bcb3aeb62c8f7948a8afa6564e3cb90680913ce not found: ID does not exist" Oct 30 06:14:26.792540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.792314 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb4126c-2c90-4dce-ac1b-95f90172634e-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:26.792540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.792351 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:26.792540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.792364 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:26.792540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.792377 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:26.839747 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.839703 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-dshm" pod "3eb4126c-2c90-4dce-ac1b-95f90172634e" (UID: "3eb4126c-2c90-4dce-ac1b-95f90172634e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:26.881637 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.881585 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb4126c-2c90-4dce-ac1b-95f90172634e-kube-api-access-25lhp" pod "3eb4126c-2c90-4dce-ac1b-95f90172634e" (UID: "3eb4126c-2c90-4dce-ac1b-95f90172634e"). InnerVolumeSpecName "kube-api-access-25lhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:26.895524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.895445 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb4126c-2c90-4dce-ac1b-95f90172634e-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:26.895846 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:26.895826 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25lhp\" (UniqueName: \"kubernetes.io/projected/3eb4126c-2c90-4dce-ac1b-95f90172634e-kube-api-access-25lhp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:27.102614 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:27.102558 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp"] Oct 30 06:14:27.121124 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:27.121089 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp"] Oct 30 06:14:27.504487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:27.500552 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-0b481b/auth-test-model-0b481b-kserve-848bd74dd9-2ggpp" podUID="3eb4126c-2c90-4dce-ac1b-95f90172634e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:14:28.617485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:28.611635 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb4126c-2c90-4dce-ac1b-95f90172634e" path="/var/lib/kubelet/pods/3eb4126c-2c90-4dce-ac1b-95f90172634e/volumes" Oct 30 06:14:30.714748 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:14:30.714711 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:14:36.026544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.026511 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr"] Oct 30 06:14:36.027483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.027445 5072 cpu_manager.go:414] "containerMap: removing container" podUID="3eb4126c-2c90-4dce-ac1b-95f90172634e" containerName="main" Oct 30 06:14:36.027483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.027483 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb4126c-2c90-4dce-ac1b-95f90172634e" containerName="main" Oct 30 06:14:36.027748 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.027723 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb4126c-2c90-4dce-ac1b-95f90172634e" containerName="main" Oct 30 06:14:36.033273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.033254 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.057073 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.057005 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-370dfa-serving-cert\"" Oct 30 06:14:36.063708 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.063678 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr"] Oct 30 06:14:36.140300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.140270 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-maas-api-tls\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.140553 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.140314 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.140629 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.140584 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-metrics-certs\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.140799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.140771 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjsn\" (UniqueName: \"kubernetes.io/projected/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-kube-api-access-4xjsn\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.242369 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.242333 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-maas-api-tls\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.242590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.242388 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.242590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.242561 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-metrics-certs\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.242727 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.242656 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjsn\" (UniqueName: \"kubernetes.io/projected/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-kube-api-access-4xjsn\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.243251 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.243169 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.245136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.245114 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-maas-api-tls\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.245243 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.245158 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-metrics-certs\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.264917 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.264886 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjsn\" (UniqueName: \"kubernetes.io/projected/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-kube-api-access-4xjsn\") pod \"maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.357532 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.350896 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:36.713498 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:14:36.713433 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a2f6c9_1a4c_4d3c_98fd_e39e30425386.slice/crio-366d2f612d2342b206924aaa60b2170a8a74089d06ba04cea39202a79186d6e4 WatchSource:0}: Error finding container 366d2f612d2342b206924aaa60b2170a8a74089d06ba04cea39202a79186d6e4: Status 404 returned error can't find the container with id 366d2f612d2342b206924aaa60b2170a8a74089d06ba04cea39202a79186d6e4 Oct 30 06:14:36.713964 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.713934 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr"] Oct 30 06:14:36.875519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:36.875483 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" event={"ID":"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386","Type":"ContainerStarted","Data":"366d2f612d2342b206924aaa60b2170a8a74089d06ba04cea39202a79186d6e4"} Oct 30 06:14:37.885069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:37.885029 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" event={"ID":"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386","Type":"ContainerStarted","Data":"16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8"} Oct 30 06:14:37.885550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:37.885144 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:37.929173 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:37.929105 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" podStartSLOduration=1.475707566 podStartE2EDuration="1.929082748s" totalImagesPullingTime="453.375182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:37.926718805 +0000 UTC m=+2637.530408065" watchObservedRunningTime="2026-10-30 06:14:37.929082748 +0000 UTC m=+2637.532772006" Oct 30 06:14:43.898400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:43.898371 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:14:50.373229 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:50.373137 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54"] Oct 30 06:14:50.373755 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:50.373472 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" podUID="2419935d-aa24-4e93-a1b4-2a20923b77df" containerName="maas-api" containerID="cri-o://d601d8a43653875d19a29926ead4e7c8621b7da52c12d4a2fd54c10fff5573dd" gracePeriod=30 Oct 30 06:14:51.112789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.111292 5072 generic.go:356] "Generic (PLEG): container finished" podID="2419935d-aa24-4e93-a1b4-2a20923b77df" containerID="d601d8a43653875d19a29926ead4e7c8621b7da52c12d4a2fd54c10fff5573dd" exitCode=0 Oct 30 06:14:51.112789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.111416 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" event={"ID":"2419935d-aa24-4e93-a1b4-2a20923b77df","Type":"ContainerDied","Data":"d601d8a43653875d19a29926ead4e7c8621b7da52c12d4a2fd54c10fff5573dd"} Oct 30 06:14:51.159235 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.159198 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 06:14:51.289061 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.289025 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2419935d-aa24-4e93-a1b4-2a20923b77df-kube-api-access-chrss\" (UniqueName: \"kubernetes.io/projected/2419935d-aa24-4e93-a1b4-2a20923b77df-kube-api-access-chrss\") pod \"2419935d-aa24-4e93-a1b4-2a20923b77df\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " Oct 30 06:14:51.289253 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.289079 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2419935d-aa24-4e93-a1b4-2a20923b77df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2419935d-aa24-4e93-a1b4-2a20923b77df-openshift-service-ca\") pod \"2419935d-aa24-4e93-a1b4-2a20923b77df\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " Oct 30 06:14:51.289298 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.289255 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls\") pod \"2419935d-aa24-4e93-a1b4-2a20923b77df\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " Oct 30 06:14:51.289344 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.289327 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-metrics-certs\") pod \"2419935d-aa24-4e93-a1b4-2a20923b77df\" (UID: \"2419935d-aa24-4e93-a1b4-2a20923b77df\") " Oct 30 06:14:51.289501 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.289475 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2419935d-aa24-4e93-a1b4-2a20923b77df-openshift-service-ca" pod "2419935d-aa24-4e93-a1b4-2a20923b77df" (UID: "2419935d-aa24-4e93-a1b4-2a20923b77df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:51.290063 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.290040 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2419935d-aa24-4e93-a1b4-2a20923b77df-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:51.291477 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.291434 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls" pod "2419935d-aa24-4e93-a1b4-2a20923b77df" (UID: "2419935d-aa24-4e93-a1b4-2a20923b77df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:51.291579 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.291475 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2419935d-aa24-4e93-a1b4-2a20923b77df-kube-api-access-chrss" pod "2419935d-aa24-4e93-a1b4-2a20923b77df" (UID: "2419935d-aa24-4e93-a1b4-2a20923b77df"). InnerVolumeSpecName "kube-api-access-chrss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:51.291579 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.291473 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-metrics-certs" pod "2419935d-aa24-4e93-a1b4-2a20923b77df" (UID: "2419935d-aa24-4e93-a1b4-2a20923b77df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:51.411110 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.411065 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:51.411691 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.411669 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2419935d-aa24-4e93-a1b4-2a20923b77df-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:51.411815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:51.411800 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chrss\" (UniqueName: \"kubernetes.io/projected/2419935d-aa24-4e93-a1b4-2a20923b77df-kube-api-access-chrss\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:14:52.138857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:52.138825 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" Oct 30 06:14:52.139105 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:52.138855 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" event={"ID":"2419935d-aa24-4e93-a1b4-2a20923b77df","Type":"ContainerDied","Data":"6b754980e43c1226e888eee1d558f32a6be2b3d1f68706c55c59e4f2f5817c44"} Oct 30 06:14:52.139105 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:52.138963 5072 scope.go:122] "RemoveContainer" containerID="d601d8a43653875d19a29926ead4e7c8621b7da52c12d4a2fd54c10fff5573dd" Oct 30 06:14:52.191685 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:52.191642 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54"] Oct 30 06:14:52.198733 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:52.198694 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54"] Oct 30 06:14:52.522376 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:52.522274 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2419935d-aa24-4e93-a1b4-2a20923b77df" path="/var/lib/kubelet/pods/2419935d-aa24-4e93-a1b4-2a20923b77df/volumes" Oct 30 06:14:53.904146 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:53.904106 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-0b481b-59b49f69d4-skq54" podUID="2419935d-aa24-4e93-a1b4-2a20923b77df" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.110:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:14:56.422790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.422754 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665"] Oct 30 06:14:56.423968 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.423947 5072 cpu_manager.go:414] "containerMap: removing container" podUID="2419935d-aa24-4e93-a1b4-2a20923b77df" containerName="maas-api" Oct 30 06:14:56.424046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.423972 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2419935d-aa24-4e93-a1b4-2a20923b77df" containerName="maas-api" Oct 30 06:14:56.424247 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.424234 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2419935d-aa24-4e93-a1b4-2a20923b77df" containerName="maas-api" Oct 30 06:14:56.430294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.430273 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.449821 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.449792 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665"] Oct 30 06:14:56.546162 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.546125 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.546353 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.546187 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.546353 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.546289 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.546353 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.546332 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns7lf\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-kube-api-access-ns7lf\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.546619 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.546600 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.546695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.546644 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81823e59-03bb-42fc-ac26-2ff5b93d3898-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.546763 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.546725 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.546824 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.546811 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.546885 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.546869 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.648825 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.648786 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81823e59-03bb-42fc-ac26-2ff5b93d3898-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.648844 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.648901 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.648963 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.649077 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.649121 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.649193 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649372 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.649239 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns7lf\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-kube-api-access-ns7lf\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.649439 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.649564 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649699 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.649675 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81823e59-03bb-42fc-ac26-2ff5b93d3898-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649803 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.649780 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.649872 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.649835 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.650107 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.650085 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.652118 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.652091 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.652256 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.652240 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.671720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.671685 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.671856 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.671840 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns7lf\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-kube-api-access-ns7lf\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:56.746601 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:56.746503 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:57.111006 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:14:57.110978 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81823e59_03bb_42fc_ac26_2ff5b93d3898.slice/crio-af594574e69e36894bf93ba1b7cdaf7862417b5f85e31b8b28793f6f87c3dbf1 WatchSource:0}: Error finding container af594574e69e36894bf93ba1b7cdaf7862417b5f85e31b8b28793f6f87c3dbf1: Status 404 returned error can't find the container with id af594574e69e36894bf93ba1b7cdaf7862417b5f85e31b8b28793f6f87c3dbf1 Oct 30 06:14:57.111993 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:57.111964 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665"] Oct 30 06:14:57.114636 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:57.114596 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:14:57.114721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:57.114688 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:14:57.114771 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:57.114729 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:14:57.185261 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:57.185226 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" event={"ID":"81823e59-03bb-42fc-ac26-2ff5b93d3898","Type":"ContainerStarted","Data":"af594574e69e36894bf93ba1b7cdaf7862417b5f85e31b8b28793f6f87c3dbf1"} Oct 30 06:14:58.270230 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:58.270191 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" event={"ID":"81823e59-03bb-42fc-ac26-2ff5b93d3898","Type":"ContainerStarted","Data":"8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8"} Oct 30 06:14:58.361557 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:58.358881 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" podStartSLOduration=2.35885572 podStartE2EDuration="2.35885572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:58.338851895 +0000 UTC m=+2657.942541152" watchObservedRunningTime="2026-10-30 06:14:58.35885572 +0000 UTC m=+2657.962544978" Oct 30 06:14:58.390522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:58.390484 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n"] Oct 30 06:14:58.390803 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:58.390754 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" containerID="cri-o://b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810" gracePeriod=30 Oct 30 06:14:58.749484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:58.749201 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:14:58.749484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:58.749372 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:14:58.749484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:58.749418 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 06:14:59.052806 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:59.052304 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:14:59.052806 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:59.052378 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:14:59.749496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:59.749217 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:14:59.749496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:14:59.749277 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 06:15:00.382639 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.382601 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt"] Oct 30 06:15:00.388386 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.388359 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.406247 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.406124 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-de2bf2-openshift-default-dockercfg-s762q\"" Oct 30 06:15:00.461480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.448369 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt"] Oct 30 06:15:00.532775 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.532744 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.533560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.533529 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.533894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.533876 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.534232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.534214 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.534649 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.534629 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.534865 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.534848 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.536964 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.536942 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.537182 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.537165 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl9q8\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-kube-api-access-xl9q8\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.537370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.537354 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.639889 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.639790 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.640210 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.640184 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.640410 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.640383 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.640592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.640564 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.640772 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.640746 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl9q8\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-kube-api-access-xl9q8\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.640959 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.640942 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.641289 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.641268 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.641431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.641415 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.645786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.645764 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.666333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.666300 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.671116 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.667744 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.671353 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.668184 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.671515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.668729 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.680556 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.669940 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.705485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.690373 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.713088 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.713016 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.729485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.726854 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.749370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.749166 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl9q8\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-kube-api-access-xl9q8\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.762490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.759129 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:15:00.777961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.777583 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 06:15:00.787618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:00.786756 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:00.982508 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:15:00.975517 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:15:01.638485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:01.637284 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt"] Oct 30 06:15:01.645103 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:15:01.645071 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fe0fd5_460a_4faf_854c_8172f9e00e2b.slice/crio-a8538169cc627e5a266b046f8a94caf5188ca4324197ab117320113b07c2b9c9 WatchSource:0}: Error finding container a8538169cc627e5a266b046f8a94caf5188ca4324197ab117320113b07c2b9c9: Status 404 returned error can't find the container with id a8538169cc627e5a266b046f8a94caf5188ca4324197ab117320113b07c2b9c9 Oct 30 06:15:01.648279 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:01.648245 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:15:01.648400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:01.648335 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:15:01.648400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:01.648386 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:15:01.747105 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:01.747060 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:15:01.747276 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:01.747128 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 06:15:02.333325 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:02.333283 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" event={"ID":"d5fe0fd5-460a-4faf-854c-8172f9e00e2b","Type":"ContainerStarted","Data":"53ece7de56395ebdce7ba495df676642dee33ee9cf0e7d865de90da2cc5a0447"} Oct 30 06:15:02.347578 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:02.347538 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" event={"ID":"d5fe0fd5-460a-4faf-854c-8172f9e00e2b","Type":"ContainerStarted","Data":"a8538169cc627e5a266b046f8a94caf5188ca4324197ab117320113b07c2b9c9"} Oct 30 06:15:02.378574 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:02.378503 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" podStartSLOduration=2.37844768 podStartE2EDuration="2.37844768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.377839788 +0000 UTC m=+2661.981529045" watchObservedRunningTime="2026-10-30 06:15:02.37844768 +0000 UTC m=+2661.982136942" Oct 30 06:15:02.747082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:02.747047 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:15:02.747290 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:02.747110 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 06:15:02.788693 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:02.788664 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:02.795048 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:02.795019 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:03.347712 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.347678 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:03.349196 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.349172 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:15:03.747745 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.747705 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:15:03.747962 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.747768 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 06:15:03.797972 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.797939 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:15:03.883267 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883226 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60af48b8-8385-45af-b091-f1788b83604e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af48b8-8385-45af-b091-f1788b83604e-istio-podinfo\") pod \"60af48b8-8385-45af-b091-f1788b83604e\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " Oct 30 06:15:03.883521 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883448 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-certs\") pod \"60af48b8-8385-45af-b091-f1788b83604e\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " Oct 30 06:15:03.883634 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883529 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60af48b8-8385-45af-b091-f1788b83604e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af48b8-8385-45af-b091-f1788b83604e-istiod-ca-cert\") pod \"60af48b8-8385-45af-b091-f1788b83604e\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " Oct 30 06:15:03.883634 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883565 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-kube-api-access-zrjsc\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-kube-api-access-zrjsc\") pod \"60af48b8-8385-45af-b091-f1788b83604e\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " Oct 30 06:15:03.883634 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883617 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-istio-token\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-istio-token\") pod \"60af48b8-8385-45af-b091-f1788b83604e\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " Oct 30 06:15:03.883822 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883644 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-data\") pod \"60af48b8-8385-45af-b091-f1788b83604e\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " Oct 30 06:15:03.883884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883827 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-socket\") pod \"60af48b8-8385-45af-b091-f1788b83604e\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " Oct 30 06:15:03.883884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883858 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-envoy\") pod \"60af48b8-8385-45af-b091-f1788b83604e\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " Oct 30 06:15:03.883884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883860 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-certs" pod "60af48b8-8385-45af-b091-f1788b83604e" (UID: "60af48b8-8385-45af-b091-f1788b83604e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:03.884068 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883900 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-credential-socket\") pod \"60af48b8-8385-45af-b091-f1788b83604e\" (UID: \"60af48b8-8385-45af-b091-f1788b83604e\") " Oct 30 06:15:03.884068 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.883946 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60af48b8-8385-45af-b091-f1788b83604e-istiod-ca-cert" pod "60af48b8-8385-45af-b091-f1788b83604e" (UID: "60af48b8-8385-45af-b091-f1788b83604e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:03.884195 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.884118 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-credential-socket" pod "60af48b8-8385-45af-b091-f1788b83604e" (UID: "60af48b8-8385-45af-b091-f1788b83604e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:03.884629 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.884302 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-socket" pod "60af48b8-8385-45af-b091-f1788b83604e" (UID: "60af48b8-8385-45af-b091-f1788b83604e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:03.884746 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.884640 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-data" pod "60af48b8-8385-45af-b091-f1788b83604e" (UID: "60af48b8-8385-45af-b091-f1788b83604e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:03.884856 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.884840 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:03.884923 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.884859 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:03.884923 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.884872 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:03.884923 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.884885 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af48b8-8385-45af-b091-f1788b83604e-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:03.884923 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.884898 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:03.895332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.894797 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60af48b8-8385-45af-b091-f1788b83604e-istio-podinfo" pod "60af48b8-8385-45af-b091-f1788b83604e" (UID: "60af48b8-8385-45af-b091-f1788b83604e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:15:03.895332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.894873 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-envoy" pod "60af48b8-8385-45af-b091-f1788b83604e" (UID: "60af48b8-8385-45af-b091-f1788b83604e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:03.895332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.894873 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-istio-token" pod "60af48b8-8385-45af-b091-f1788b83604e" (UID: "60af48b8-8385-45af-b091-f1788b83604e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:03.897982 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.897940 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-kube-api-access-zrjsc" pod "60af48b8-8385-45af-b091-f1788b83604e" (UID: "60af48b8-8385-45af-b091-f1788b83604e"). InnerVolumeSpecName "kube-api-access-zrjsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:03.986032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.985998 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af48b8-8385-45af-b091-f1788b83604e-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:03.986032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.986027 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af48b8-8385-45af-b091-f1788b83604e-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:03.986032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.986038 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrjsc\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-kube-api-access-zrjsc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:03.986262 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:03.986049 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60af48b8-8385-45af-b091-f1788b83604e-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:04.359566 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.359529 5072 generic.go:356] "Generic (PLEG): container finished" podID="60af48b8-8385-45af-b091-f1788b83604e" containerID="b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810" exitCode=0 Oct 30 06:15:04.360069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.359606 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" Oct 30 06:15:04.360069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.359610 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" event={"ID":"60af48b8-8385-45af-b091-f1788b83604e","Type":"ContainerDied","Data":"b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810"} Oct 30 06:15:04.360069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.359741 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n" event={"ID":"60af48b8-8385-45af-b091-f1788b83604e","Type":"ContainerDied","Data":"8d14a4a35006d117e5de4dce6d809d631cf903aa188393bcac7de61f44cc96f8"} Oct 30 06:15:04.360069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.359775 5072 scope.go:122] "RemoveContainer" containerID="b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810" Oct 30 06:15:04.373859 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.373836 5072 scope.go:122] "RemoveContainer" containerID="b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810" Oct 30 06:15:04.374207 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:15:04.374177 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810\": container with ID starting with b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810 not found: ID does not exist" containerID="b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810" Oct 30 06:15:04.374332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.374217 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810"} err="failed to get container status \"b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810\": rpc error: code = NotFound desc = could not find container \"b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810\": container with ID starting with b1a81ed7bb76297d11aa5e50a1eba90dc4e99b56be9626741eb0ea71d94f2810 not found: ID does not exist" Oct 30 06:15:04.422027 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.421855 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n"] Oct 30 06:15:04.432450 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.432413 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-rrl8n"] Oct 30 06:15:04.523558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.523527 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60af48b8-8385-45af-b091-f1788b83604e" path="/var/lib/kubelet/pods/60af48b8-8385-45af-b091-f1788b83604e/volumes" Oct 30 06:15:04.753437 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.753343 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:15:04.755709 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.755683 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:15:04.755869 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:04.755768 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:15:30.711802 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:15:30.711762 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:15:39.266479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.266424 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc"] Oct 30 06:15:39.267364 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.267320 5072 cpu_manager.go:414] "containerMap: removing container" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" Oct 30 06:15:39.267364 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.267344 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" Oct 30 06:15:39.267646 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.267630 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="60af48b8-8385-45af-b091-f1788b83604e" containerName="istio-proxy" Oct 30 06:15:39.272468 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.272439 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.287928 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.287880 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fb9345-openshift-default-dockercfg-w4p9m\"" Oct 30 06:15:39.305186 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.305149 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc"] Oct 30 06:15:39.449483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.448449 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854b69be-96e5-4912-8ba9-1df7ce008e7c-istiod-ca-cert\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.449483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.448602 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-token\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.449483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.448649 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-socket\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.449483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.448685 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-envoy\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.449483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.448715 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-podinfo\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.449483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.448755 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-data\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.449483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.448818 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-credential-socket\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.449483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.448872 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlm4z\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-kube-api-access-mlm4z\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.449483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.448966 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-certs\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.564540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.560957 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-token\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.564991 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.564964 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-socket\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.574499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.565167 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-envoy\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.583910 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.583879 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-podinfo\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.584207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.584184 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-data\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.584390 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.584374 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-credential-socket\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.584560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.584544 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlm4z\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-kube-api-access-mlm4z\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.584765 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.584748 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-certs\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.586521 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.586497 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854b69be-96e5-4912-8ba9-1df7ce008e7c-istiod-ca-cert\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.598685 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.581843 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-socket\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.599058 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.585708 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-credential-socket\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.599227 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.586221 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-certs\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.599327 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.585091 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-data\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.599431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.587259 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854b69be-96e5-4912-8ba9-1df7ce008e7c-istiod-ca-cert\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.644997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.644001 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-podinfo\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.670068 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.670036 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-envoy\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.676078 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.675989 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-token\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.677699 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.677672 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlm4z\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-kube-api-access-mlm4z\") pod \"e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:39.891575 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:39.891543 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:40.228229 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:15:40.228200 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854b69be_96e5_4912_8ba9_1df7ce008e7c.slice/crio-27d55b0243fc63f2dbf09ba0886cab1e9858280535a28bc550e316f5acc27623 WatchSource:0}: Error finding container 27d55b0243fc63f2dbf09ba0886cab1e9858280535a28bc550e316f5acc27623: Status 404 returned error can't find the container with id 27d55b0243fc63f2dbf09ba0886cab1e9858280535a28bc550e316f5acc27623 Oct 30 06:15:40.229021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:40.228993 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc"] Oct 30 06:15:40.231898 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:40.231865 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:15:40.232011 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:40.231935 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:15:40.232011 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:40.231972 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:15:40.754539 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:40.754489 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" event={"ID":"854b69be-96e5-4912-8ba9-1df7ce008e7c","Type":"ContainerStarted","Data":"95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34"} Oct 30 06:15:40.755047 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:40.754565 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" event={"ID":"854b69be-96e5-4912-8ba9-1df7ce008e7c","Type":"ContainerStarted","Data":"27d55b0243fc63f2dbf09ba0886cab1e9858280535a28bc550e316f5acc27623"} Oct 30 06:15:40.819355 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:40.819280 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" podStartSLOduration=1.819257701 podStartE2EDuration="1.819257701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:40.811694722 +0000 UTC m=+2700.415383980" watchObservedRunningTime="2026-10-30 06:15:40.819257701 +0000 UTC m=+2700.422946959" Oct 30 06:15:40.892592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:40.892557 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:41.899265 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:41.899235 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:42.770327 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:42.770285 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:42.771168 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:42.771151 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:15:48.138908 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:48.138872 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:15:48.147740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:48.147717 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:15:50.494329 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:50.494294 5072 scope.go:122] "RemoveContainer" containerID="84bbc47a1e50caeed4f634f180494c9094dc43182d5d93739b98f90eba17bd1f" Oct 30 06:15:52.924482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:52.920097 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2"] Oct 30 06:15:52.952647 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:52.933647 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2"] Oct 30 06:15:52.952647 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:52.933767 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:52.968699 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:52.963200 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-de2bf2-serving-cert\"" Oct 30 06:15:53.075507 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.067680 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8nf7\" (UniqueName: \"kubernetes.io/projected/d02b2f01-c051-4989-bbc8-f9d174762e2f-kube-api-access-n8nf7\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.075507 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.067845 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-metrics-certs\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.075507 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.067923 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.075507 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.068110 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02b2f01-c051-4989-bbc8-f9d174762e2f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.175531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.169516 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02b2f01-c051-4989-bbc8-f9d174762e2f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.175531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.169636 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8nf7\" (UniqueName: \"kubernetes.io/projected/d02b2f01-c051-4989-bbc8-f9d174762e2f-kube-api-access-n8nf7\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.175531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.169758 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-metrics-certs\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.175531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.169833 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.175922 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.175773 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02b2f01-c051-4989-bbc8-f9d174762e2f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.214443 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.214407 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-metrics-certs\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.235840 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.235791 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8nf7\" (UniqueName: \"kubernetes.io/projected/d02b2f01-c051-4989-bbc8-f9d174762e2f-kube-api-access-n8nf7\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.253189 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.253156 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:53.323942 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.323827 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:54.003549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:53.994648 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2"] Oct 30 06:15:54.146482 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:15:54.134897 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02b2f01_c051_4989_bbc8_f9d174762e2f.slice/crio-21fd878eca69d50b7e310b58e4a42007f7c186d9d756527fa84ac4b81d46b619 WatchSource:0}: Error finding container 21fd878eca69d50b7e310b58e4a42007f7c186d9d756527fa84ac4b81d46b619: Status 404 returned error can't find the container with id 21fd878eca69d50b7e310b58e4a42007f7c186d9d756527fa84ac4b81d46b619 Oct 30 06:15:54.171319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:54.152570 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:15:54.268903 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:54.268810 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" event={"ID":"d02b2f01-c051-4989-bbc8-f9d174762e2f","Type":"ContainerStarted","Data":"21fd878eca69d50b7e310b58e4a42007f7c186d9d756527fa84ac4b81d46b619"} Oct 30 06:15:55.278422 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:55.278390 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" event={"ID":"d02b2f01-c051-4989-bbc8-f9d174762e2f","Type":"ContainerStarted","Data":"e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332"} Oct 30 06:15:55.278877 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:55.278637 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:15:55.318765 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:55.318718 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" podStartSLOduration=2.807149672 podStartE2EDuration="3.318704044s" totalImagesPullingTime="511.554372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:55.317608882 +0000 UTC m=+2714.921298161" watchObservedRunningTime="2026-10-30 06:15:55.318704044 +0000 UTC m=+2714.922393300" Oct 30 06:15:56.099582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.099521 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8"] Oct 30 06:15:56.099787 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.099739 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" podUID="ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" containerName="maas-api" containerID="cri-o://51bb7cb0d62866387eeb95d5f24e4aaad8bdd3b096b9f1330124d3e82b372d6e" gracePeriod=30 Oct 30 06:15:56.289264 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.289222 5072 generic.go:356] "Generic (PLEG): container finished" podID="ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" containerID="51bb7cb0d62866387eeb95d5f24e4aaad8bdd3b096b9f1330124d3e82b372d6e" exitCode=0 Oct 30 06:15:56.289697 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.289290 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" event={"ID":"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f","Type":"ContainerDied","Data":"51bb7cb0d62866387eeb95d5f24e4aaad8bdd3b096b9f1330124d3e82b372d6e"} Oct 30 06:15:56.769490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.757647 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 06:15:56.876487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.875815 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-kube-api-access-p6cw6\" (UniqueName: \"kubernetes.io/projected/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-kube-api-access-p6cw6\") pod \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " Oct 30 06:15:56.876487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.876011 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-openshift-service-ca\") pod \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " Oct 30 06:15:56.876487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.876045 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-maas-api-tls\") pod \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " Oct 30 06:15:56.876487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.876212 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-metrics-certs\") pod \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\" (UID: \"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f\") " Oct 30 06:15:56.885046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.885014 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-openshift-service-ca" pod "ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" (UID: "ad0e2546-7801-44aa-a59a-d3fdc77a6d5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:56.888128 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.888094 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-maas-api-tls" pod "ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" (UID: "ad0e2546-7801-44aa-a59a-d3fdc77a6d5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:56.888450 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.888429 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-kube-api-access-p6cw6" pod "ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" (UID: "ad0e2546-7801-44aa-a59a-d3fdc77a6d5f"). InnerVolumeSpecName "kube-api-access-p6cw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:56.891250 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.891227 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-metrics-certs" pod "ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" (UID: "ad0e2546-7801-44aa-a59a-d3fdc77a6d5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:56.979164 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.979088 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:56.979164 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.979121 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:56.979164 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.979130 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:56.979164 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:56.979138 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6cw6\" (UniqueName: \"kubernetes.io/projected/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f-kube-api-access-p6cw6\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:15:57.298387 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:57.298300 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" event={"ID":"ad0e2546-7801-44aa-a59a-d3fdc77a6d5f","Type":"ContainerDied","Data":"699cc5ece9b9d370be173f4b1588f6e562e0eeb0e48094fec2550e1e6749ac92"} Oct 30 06:15:57.298387 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:57.298366 5072 scope.go:122] "RemoveContainer" containerID="51bb7cb0d62866387eeb95d5f24e4aaad8bdd3b096b9f1330124d3e82b372d6e" Oct 30 06:15:57.298387 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:57.298315 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8" Oct 30 06:15:57.374559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:57.374527 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8"] Oct 30 06:15:57.379421 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:57.379390 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b4ca14-66df578c-j68c8"] Oct 30 06:15:58.521786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:15:58.521750 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" path="/var/lib/kubelet/pods/ad0e2546-7801-44aa-a59a-d3fdc77a6d5f/volumes" Oct 30 06:16:00.719491 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:16:00.719281 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:16:01.295317 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:01.295286 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:16:03.890586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:03.890549 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj"] Oct 30 06:16:03.891010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:03.890795 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" containerID="cri-o://69496c53bbe91993a6b71a0753ca0a5b3c63b621445fdbf75cba475aec03eb5d" gracePeriod=30 Oct 30 06:16:09.343557 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.343427 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx"] Oct 30 06:16:09.408723 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.408692 5072 generic.go:356] "Generic (PLEG): container finished" podID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerID="69496c53bbe91993a6b71a0753ca0a5b3c63b621445fdbf75cba475aec03eb5d" exitCode=0 Oct 30 06:16:09.408867 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.408701 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" event={"ID":"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0","Type":"ContainerDied","Data":"69496c53bbe91993a6b71a0753ca0a5b3c63b621445fdbf75cba475aec03eb5d"} Oct 30 06:16:09.733735 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.733709 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:16:09.873403 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.873364 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-kube-api-access-jvpzq\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-kube-api-access-jvpzq\") pod \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " Oct 30 06:16:09.873650 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.873565 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-token\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-token\") pod \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " Oct 30 06:16:09.873731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.873648 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-certs\") pod \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " Oct 30 06:16:09.873731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.873725 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istiod-ca-cert\") pod \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " Oct 30 06:16:09.873859 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.873758 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-socket\") pod \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " Oct 30 06:16:09.873859 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.873841 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-envoy\") pod \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " Oct 30 06:16:09.873971 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.873886 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-podinfo\") pod \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " Oct 30 06:16:09.873971 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.873940 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-data\") pod \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " Oct 30 06:16:09.874074 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.873973 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-credential-socket\") pod \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\" (UID: \"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0\") " Oct 30 06:16:09.874074 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.874004 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-certs" pod "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" (UID: "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.874605 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.874578 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-data" pod "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" (UID: "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.874752 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.874611 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.874853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.874831 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-credential-socket" pod "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" (UID: "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.874919 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.874875 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istiod-ca-cert" pod "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" (UID: "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:09.875039 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.875010 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-socket" pod "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" (UID: "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.877335 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.877080 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-podinfo" pod "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" (UID: "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:16:09.877497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.877441 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-envoy" pod "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" (UID: "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.877935 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.877894 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-kube-api-access-jvpzq" pod "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" (UID: "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0"). InnerVolumeSpecName "kube-api-access-jvpzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:09.878309 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.878277 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-token" pod "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" (UID: "784081e5-9fbc-4a1a-9763-1fd9d9cf33a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:09.976726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.976651 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.976726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.976682 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.976726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.976694 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.976726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.976706 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.976726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.976719 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.977026 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.976730 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.977026 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.976742 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.977026 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:09.976754 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpzq\" (UniqueName: \"kubernetes.io/projected/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0-kube-api-access-jvpzq\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:10.419138 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:10.419107 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" Oct 30 06:16:10.419688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:10.419106 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj" event={"ID":"784081e5-9fbc-4a1a-9763-1fd9d9cf33a0","Type":"ContainerDied","Data":"dd561decb53f58bc1113b70ef350fbd5b936343847c3e24867ce25e593786794"} Oct 30 06:16:10.419688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:10.419299 5072 scope.go:122] "RemoveContainer" containerID="69496c53bbe91993a6b71a0753ca0a5b3c63b621445fdbf75cba475aec03eb5d" Oct 30 06:16:10.468738 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:10.468700 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj"] Oct 30 06:16:10.479488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:10.479438 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-zk2sj"] Oct 30 06:16:10.523888 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:10.523859 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" path="/var/lib/kubelet/pods/784081e5-9fbc-4a1a-9763-1fd9d9cf33a0/volumes" Oct 30 06:16:24.371648 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.371585 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" containerID="cri-o://cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6" gracePeriod=45 Oct 30 06:16:24.553800 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.553490 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Oct 30 06:16:24.771010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.770979 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:16:24.904032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.903992 5072 generic.go:356] "Generic (PLEG): container finished" podID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerID="cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6" exitCode=0 Oct 30 06:16:24.904241 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.904046 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" Oct 30 06:16:24.904241 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.904068 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" event={"ID":"4285a424-8b5e-4aa1-b77e-3ad6fe415b62","Type":"ContainerDied","Data":"cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6"} Oct 30 06:16:24.904241 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.904121 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx" event={"ID":"4285a424-8b5e-4aa1-b77e-3ad6fe415b62","Type":"ContainerDied","Data":"731a249418a8f3bde4e9bbbfcd419f2b629b1c4b1dab1076e9b8d8c6eebccc47"} Oct 30 06:16:24.904241 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.904144 5072 scope.go:122] "RemoveContainer" containerID="cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6" Oct 30 06:16:24.951056 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.950956 5072 scope.go:122] "RemoveContainer" containerID="cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6" Oct 30 06:16:24.951591 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:16:24.951518 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6\": container with ID starting with cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6 not found: ID does not exist" containerID="cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6" Oct 30 06:16:24.951591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.951556 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6"} err="failed to get container status \"cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6\": rpc error: code = NotFound desc = could not find container \"cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6\": container with ID starting with cda80edc7dd0242fb71471cd4dab9b3a6edae438f928d843ffd00e8f44db56c6 not found: ID does not exist" Oct 30 06:16:24.965669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.965222 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tmp-dir\") pod \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " Oct 30 06:16:24.965669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.965327 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-model-cache\") pod \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " Oct 30 06:16:24.965669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.965443 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tls-certs\" (UniqueName: \"kubernetes.io/secret/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tls-certs\") pod \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " Oct 30 06:16:24.965669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.965501 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-home\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-home\") pod \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " Oct 30 06:16:24.965669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.965576 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-dshm\") pod \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " Oct 30 06:16:25.000926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.966036 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-kube-api-access-lt5pz\" (UniqueName: \"kubernetes.io/projected/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-kube-api-access-lt5pz\") pod \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\" (UID: \"4285a424-8b5e-4aa1-b77e-3ad6fe415b62\") " Oct 30 06:16:25.000926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.967743 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-home" pod "4285a424-8b5e-4aa1-b77e-3ad6fe415b62" (UID: "4285a424-8b5e-4aa1-b77e-3ad6fe415b62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:25.000926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.967863 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tmp-dir" pod "4285a424-8b5e-4aa1-b77e-3ad6fe415b62" (UID: "4285a424-8b5e-4aa1-b77e-3ad6fe415b62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:25.000926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:24.968619 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-model-cache" pod "4285a424-8b5e-4aa1-b77e-3ad6fe415b62" (UID: "4285a424-8b5e-4aa1-b77e-3ad6fe415b62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:25.006529 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.003736 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tls-certs" pod "4285a424-8b5e-4aa1-b77e-3ad6fe415b62" (UID: "4285a424-8b5e-4aa1-b77e-3ad6fe415b62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:25.006529 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.005412 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-kube-api-access-lt5pz" pod "4285a424-8b5e-4aa1-b77e-3ad6fe415b62" (UID: "4285a424-8b5e-4aa1-b77e-3ad6fe415b62"). InnerVolumeSpecName "kube-api-access-lt5pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:25.006912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.006865 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-dshm" pod "4285a424-8b5e-4aa1-b77e-3ad6fe415b62" (UID: "4285a424-8b5e-4aa1-b77e-3ad6fe415b62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:25.068955 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.068924 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:25.068955 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.068952 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:25.068955 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.068961 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:25.069226 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.068970 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt5pz\" (UniqueName: \"kubernetes.io/projected/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-kube-api-access-lt5pz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:25.069226 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.068999 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:25.069226 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.069010 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4285a424-8b5e-4aa1-b77e-3ad6fe415b62-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:16:25.282676 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.282644 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx"] Oct 30 06:16:25.292966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:25.292933 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-2-w4-698ab2-kserve-68d4bc988-jsrnx"] Oct 30 06:16:26.547792 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:26.547758 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" path="/var/lib/kubelet/pods/4285a424-8b5e-4aa1-b77e-3ad6fe415b62/volumes" Oct 30 06:16:30.720092 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:16:30.720045 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:16:30.750406 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.750368 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf"] Oct 30 06:16:30.751915 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.751883 5072 cpu_manager.go:414] "containerMap: removing container" podUID="ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" containerName="maas-api" Oct 30 06:16:30.752095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.752077 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" containerName="maas-api" Oct 30 06:16:30.752311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.752265 5072 cpu_manager.go:414] "containerMap: removing container" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" Oct 30 06:16:30.752311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.752285 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" Oct 30 06:16:30.752311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.752305 5072 cpu_manager.go:414] "containerMap: removing container" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" Oct 30 06:16:30.752311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.752313 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" Oct 30 06:16:30.752706 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.752520 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="4285a424-8b5e-4aa1-b77e-3ad6fe415b62" containerName="main" Oct 30 06:16:30.752706 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.752540 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="784081e5-9fbc-4a1a-9763-1fd9d9cf33a0" containerName="istio-proxy" Oct 30 06:16:30.752706 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.752558 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0e2546-7801-44aa-a59a-d3fdc77a6d5f" containerName="maas-api" Oct 30 06:16:30.759577 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.759551 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.871679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.834511 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.871679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.834564 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.871679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.834602 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.871679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.834677 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.871679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.834785 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.871679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.834834 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.871679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.834942 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2g5\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-kube-api-access-9k2g5\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.871679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.835016 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.871679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.835043 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:30.901222 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:30.901165 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf"] Oct 30 06:16:31.042537 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.041797 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.042537 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.041980 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.042537 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.042433 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2g5\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-kube-api-access-9k2g5\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.046134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.042824 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.046134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.042966 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.046134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.043437 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.046134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.043575 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.046134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.043752 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.046134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.044049 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.046134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.045271 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.046792 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.046597 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.046978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.046929 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.057566 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.048020 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.057566 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.048905 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.084863 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.083098 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.089426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.089392 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.090571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.090374 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.091346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.091323 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2g5\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-kube-api-access-9k2g5\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.224044 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.224003 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t"] Oct 30 06:16:31.231963 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.231548 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.245403 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.245259 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fb9345-serving-cert\"" Oct 30 06:16:31.250312 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.249411 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t"] Oct 30 06:16:31.257199 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.256044 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:31.353528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.351821 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvkzh\" (UniqueName: \"kubernetes.io/projected/9a517262-378e-400c-817d-3b72b61ac3ee-kube-api-access-mvkzh\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.353528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.351969 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a517262-378e-400c-817d-3b72b61ac3ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.353528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.352089 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.353528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.352160 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.454584 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.454545 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a517262-378e-400c-817d-3b72b61ac3ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.454844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.454682 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.454844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.454749 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.455106 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.454913 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvkzh\" (UniqueName: \"kubernetes.io/projected/9a517262-378e-400c-817d-3b72b61ac3ee-kube-api-access-mvkzh\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.455739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.455344 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a517262-378e-400c-817d-3b72b61ac3ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.455739 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:16:31.455408 5072 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-serving-cert: secret "maas-api-e2e-ipp-a-fb9345-serving-cert" not found Oct 30 06:16:31.455739 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:16:31.455524 5072 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls podName:9a517262-378e-400c-817d-3b72b61ac3ee nodeName:}" failed. No retries permitted until 2026-10-30 06:16:31.95550522 +0000 UTC m=+2751.559194470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls") pod "maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" (UID: "9a517262-378e-400c-817d-3b72b61ac3ee") : secret "maas-api-e2e-ipp-a-fb9345-serving-cert" not found Oct 30 06:16:31.458602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.458576 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.480092 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.480050 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvkzh\" (UniqueName: \"kubernetes.io/projected/9a517262-378e-400c-817d-3b72b61ac3ee-kube-api-access-mvkzh\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.585183 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.585148 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf"] Oct 30 06:16:31.586746 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:16:31.586717 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567a0a2d_a294_4b2d_9a0b_ddc235184e72.slice/crio-ca9d1bcda9e020073d274c0dbf6c69d8fb37faef12b545ec6e57a01ed70d106f WatchSource:0}: Error finding container ca9d1bcda9e020073d274c0dbf6c69d8fb37faef12b545ec6e57a01ed70d106f: Status 404 returned error can't find the container with id ca9d1bcda9e020073d274c0dbf6c69d8fb37faef12b545ec6e57a01ed70d106f Oct 30 06:16:31.589760 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.589728 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:16:31.589857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.589795 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:16:31.589857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.589826 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:16:31.968652 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.968619 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:31.972717 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:31.972687 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:32.100011 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:32.099966 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" event={"ID":"567a0a2d-a294-4b2d-9a0b-ddc235184e72","Type":"ContainerStarted","Data":"127f4f650bebfbe066e7f2b34a5ffdcce6544397adf5a5ba3de38df29fa80119"} Oct 30 06:16:32.100211 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:32.100034 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" event={"ID":"567a0a2d-a294-4b2d-9a0b-ddc235184e72","Type":"ContainerStarted","Data":"ca9d1bcda9e020073d274c0dbf6c69d8fb37faef12b545ec6e57a01ed70d106f"} Oct 30 06:16:32.147269 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:32.147202 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" podStartSLOduration=2.147179823 podStartE2EDuration="2.147179823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:32.143120527 +0000 UTC m=+2751.746809780" watchObservedRunningTime="2026-10-30 06:16:32.147179823 +0000 UTC m=+2751.750869092" Oct 30 06:16:32.153307 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:32.153278 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:32.268516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:32.267548 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:32.268678 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:32.268518 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 06:16:32.268678 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:32.268580 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 06:16:32.484280 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:32.484253 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t"] Oct 30 06:16:32.484752 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:16:32.484729 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a517262_378e_400c_817d_3b72b61ac3ee.slice/crio-e09bb26bb0fdcdd47df3a7e828c98a0eb46b42279fde646cb4f1353aec65bd76 WatchSource:0}: Error finding container e09bb26bb0fdcdd47df3a7e828c98a0eb46b42279fde646cb4f1353aec65bd76: Status 404 returned error can't find the container with id e09bb26bb0fdcdd47df3a7e828c98a0eb46b42279fde646cb4f1353aec65bd76 Oct 30 06:16:33.149494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.148687 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" event={"ID":"9a517262-378e-400c-817d-3b72b61ac3ee","Type":"ContainerStarted","Data":"e09bb26bb0fdcdd47df3a7e828c98a0eb46b42279fde646cb4f1353aec65bd76"} Oct 30 06:16:33.151178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.151137 5072 generic.go:356] "Generic (PLEG): container finished" podID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerID="53ece7de56395ebdce7ba495df676642dee33ee9cf0e7d865de90da2cc5a0447" exitCode=0 Oct 30 06:16:33.154296 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.152839 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" event={"ID":"d5fe0fd5-460a-4faf-854c-8172f9e00e2b","Type":"ContainerDied","Data":"53ece7de56395ebdce7ba495df676642dee33ee9cf0e7d865de90da2cc5a0447"} Oct 30 06:16:33.154434 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.154349 5072 scope.go:122] "RemoveContainer" containerID="53ece7de56395ebdce7ba495df676642dee33ee9cf0e7d865de90da2cc5a0447" Oct 30 06:16:33.157699 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.157665 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:16:33.157894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.157783 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:16:33.157894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.157833 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:16:33.282491 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.277722 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 06:16:33.282904 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.282867 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 06:16:33.357486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:33.353026 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:16:34.163337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:34.163298 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" event={"ID":"9a517262-378e-400c-817d-3b72b61ac3ee","Type":"ContainerStarted","Data":"77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181"} Oct 30 06:16:34.163844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:34.163557 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:34.166555 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:34.166526 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" event={"ID":"d5fe0fd5-460a-4faf-854c-8172f9e00e2b","Type":"ContainerStarted","Data":"d1dfe6f5c9e86cab69221ca26d0bbf2d96d6d30a33214b229cd3c0173e460229"} Oct 30 06:16:34.202746 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:34.202698 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" podStartSLOduration=2.6683225459999997 podStartE2EDuration="3.202683236s" totalImagesPullingTime="534.36069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:34.202420408 +0000 UTC m=+2753.806109670" watchObservedRunningTime="2026-10-30 06:16:34.202683236 +0000 UTC m=+2753.806372492" Oct 30 06:16:34.257525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:34.257488 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 06:16:34.257707 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:34.257560 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 06:16:34.788354 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:34.788322 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:16:34.789031 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:34.789010 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 06:16:34.789194 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:34.789171 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 06:16:35.259486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:35.259050 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 06:16:35.259486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:35.259138 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 06:16:35.788969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:35.788930 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 06:16:35.789176 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:35.789004 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 06:16:36.258054 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:36.258015 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 06:16:36.259269 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:36.259235 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 06:16:36.788676 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:36.788630 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 06:16:36.788676 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:36.788678 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 06:16:37.274940 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:37.264352 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 06:16:37.275306 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:37.275268 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 06:16:37.788764 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:37.788727 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 06:16:37.789201 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:37.788778 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 06:16:38.259151 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:38.258700 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 06:16:38.259151 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:38.258765 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 06:16:38.788612 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:38.788574 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 06:16:38.788938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:38.788907 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 06:16:39.478385 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:39.478342 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:39.479149 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:39.479126 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:39.480898 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:39.480869 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:16:39.788014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:39.787931 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 06:16:39.788014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:39.787983 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 06:16:40.199731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:40.199703 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:16:40.788559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:40.788524 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 06:16:40.788734 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:40.788582 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 06:16:41.793521 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:41.793447 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:16:41.793912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:41.793827 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:16:41.794445 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:16:41.794428 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:17:00.713596 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:17:00.713555 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:17:22.506633 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:22.506600 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb"] Oct 30 06:17:22.507025 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:22.506825 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" podUID="b922baf8-fe45-4754-a3fd-7d604a4dee19" containerName="maas-api" containerID="cri-o://17b67e03869fd47810260bf91d07cd6cb8b6d24832a64eabb5db27e885791b63" gracePeriod=30 Oct 30 06:17:23.070392 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.065258 5072 generic.go:356] "Generic (PLEG): container finished" podID="b922baf8-fe45-4754-a3fd-7d604a4dee19" containerID="17b67e03869fd47810260bf91d07cd6cb8b6d24832a64eabb5db27e885791b63" exitCode=0 Oct 30 06:17:23.070392 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.065505 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" event={"ID":"b922baf8-fe45-4754-a3fd-7d604a4dee19","Type":"ContainerDied","Data":"17b67e03869fd47810260bf91d07cd6cb8b6d24832a64eabb5db27e885791b63"} Oct 30 06:17:23.144702 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.144673 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 06:17:23.250483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.250421 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-metrics-certs\") pod \"b922baf8-fe45-4754-a3fd-7d604a4dee19\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " Oct 30 06:17:23.250810 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.250783 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b922baf8-fe45-4754-a3fd-7d604a4dee19-kube-api-access-xnpzz\" (UniqueName: \"kubernetes.io/projected/b922baf8-fe45-4754-a3fd-7d604a4dee19-kube-api-access-xnpzz\") pod \"b922baf8-fe45-4754-a3fd-7d604a4dee19\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " Oct 30 06:17:23.251231 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.251209 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-maas-api-tls\") pod \"b922baf8-fe45-4754-a3fd-7d604a4dee19\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " Oct 30 06:17:23.251544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.251525 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b922baf8-fe45-4754-a3fd-7d604a4dee19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922baf8-fe45-4754-a3fd-7d604a4dee19-openshift-service-ca\") pod \"b922baf8-fe45-4754-a3fd-7d604a4dee19\" (UID: \"b922baf8-fe45-4754-a3fd-7d604a4dee19\") " Oct 30 06:17:23.294512 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.291425 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b922baf8-fe45-4754-a3fd-7d604a4dee19-openshift-service-ca" pod "b922baf8-fe45-4754-a3fd-7d604a4dee19" (UID: "b922baf8-fe45-4754-a3fd-7d604a4dee19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:23.294512 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.292734 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b922baf8-fe45-4754-a3fd-7d604a4dee19-kube-api-access-xnpzz" pod "b922baf8-fe45-4754-a3fd-7d604a4dee19" (UID: "b922baf8-fe45-4754-a3fd-7d604a4dee19"). InnerVolumeSpecName "kube-api-access-xnpzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:23.294512 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.293235 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-maas-api-tls" pod "b922baf8-fe45-4754-a3fd-7d604a4dee19" (UID: "b922baf8-fe45-4754-a3fd-7d604a4dee19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:23.338757 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.337622 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:23.338757 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.337664 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b922baf8-fe45-4754-a3fd-7d604a4dee19-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:23.338757 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.337682 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnpzz\" (UniqueName: \"kubernetes.io/projected/b922baf8-fe45-4754-a3fd-7d604a4dee19-kube-api-access-xnpzz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:23.353477 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.347928 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-metrics-certs" pod "b922baf8-fe45-4754-a3fd-7d604a4dee19" (UID: "b922baf8-fe45-4754-a3fd-7d604a4dee19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:23.439076 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:23.439040 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b922baf8-fe45-4754-a3fd-7d604a4dee19-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:24.106222 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:24.106172 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" event={"ID":"b922baf8-fe45-4754-a3fd-7d604a4dee19","Type":"ContainerDied","Data":"86f09591f4a9768edffff3f50ab2fd1b7a33dfb87c787ea2c9bb5b7ca58a9ca9"} Oct 30 06:17:24.106960 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:24.106936 5072 scope.go:122] "RemoveContainer" containerID="17b67e03869fd47810260bf91d07cd6cb8b6d24832a64eabb5db27e885791b63" Oct 30 06:17:24.107294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:24.107276 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb" Oct 30 06:17:24.168197 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:24.168164 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb"] Oct 30 06:17:24.178243 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:24.178215 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-698ab2-7458d57998-rkczb"] Oct 30 06:17:24.524427 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:24.524333 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b922baf8-fe45-4754-a3fd-7d604a4dee19" path="/var/lib/kubelet/pods/b922baf8-fe45-4754-a3fd-7d604a4dee19/volumes" Oct 30 06:17:29.010481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:29.008606 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb"] Oct 30 06:17:29.010481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:29.008887 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" containerID="cri-o://87d078dd3a579c1fac2e93b53a5bdbbb5218b6d982beecf02c3f72eec7512206" gracePeriod=30 Oct 30 06:17:29.084531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:29.083392 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l"] Oct 30 06:17:29.084531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:29.083691 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" containerID="cri-o://60522e0d0c6e8b19e23b165632614349fe8b7a09a9e75c96db9388a007032c81" gracePeriod=30 Oct 30 06:17:31.132485 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:17:31.108321 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:17:31.796678 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.796642 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn"] Oct 30 06:17:31.797957 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.797933 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b922baf8-fe45-4754-a3fd-7d604a4dee19" containerName="maas-api" Oct 30 06:17:31.798080 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.797961 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b922baf8-fe45-4754-a3fd-7d604a4dee19" containerName="maas-api" Oct 30 06:17:31.798173 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.798157 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b922baf8-fe45-4754-a3fd-7d604a4dee19" containerName="maas-api" Oct 30 06:17:31.803758 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.803736 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.827496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.827422 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn"] Oct 30 06:17:31.827661 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.827590 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-07b278-serving-cert\"" Oct 30 06:17:31.846689 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.846654 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84881af6-829e-4489-9c91-319b111ef962-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.846916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.846894 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvlz\" (UniqueName: \"kubernetes.io/projected/84881af6-829e-4489-9c91-319b111ef962-kube-api-access-gqvlz\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.847055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.846943 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-metrics-certs\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.847120 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.847092 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-maas-api-tls\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.949097 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.948901 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-maas-api-tls\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.950303 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.949567 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84881af6-829e-4489-9c91-319b111ef962-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.950303 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.949641 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvlz\" (UniqueName: \"kubernetes.io/projected/84881af6-829e-4489-9c91-319b111ef962-kube-api-access-gqvlz\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.950303 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.949685 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-metrics-certs\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.953408 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.953381 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-metrics-certs\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.954316 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.954291 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-maas-api-tls\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.955007 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.954985 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84881af6-829e-4489-9c91-319b111ef962-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:31.973439 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:31.973413 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvlz\" (UniqueName: \"kubernetes.io/projected/84881af6-829e-4489-9c91-319b111ef962-kube-api-access-gqvlz\") pod \"maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:32.136472 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:32.136423 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:32.458787 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:32.458756 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn"] Oct 30 06:17:32.459189 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:17:32.459163 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84881af6_829e_4489_9c91_319b111ef962.slice/crio-89001d7dfc4eb649863463b775c26f46bf9129eba2c1a229770d1a6a335d0ee4 WatchSource:0}: Error finding container 89001d7dfc4eb649863463b775c26f46bf9129eba2c1a229770d1a6a335d0ee4: Status 404 returned error can't find the container with id 89001d7dfc4eb649863463b775c26f46bf9129eba2c1a229770d1a6a335d0ee4 Oct 30 06:17:33.340046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:33.340007 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" event={"ID":"84881af6-829e-4489-9c91-319b111ef962","Type":"ContainerStarted","Data":"da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc"} Oct 30 06:17:33.340585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:33.340072 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" event={"ID":"84881af6-829e-4489-9c91-319b111ef962","Type":"ContainerStarted","Data":"89001d7dfc4eb649863463b775c26f46bf9129eba2c1a229770d1a6a335d0ee4"} Oct 30 06:17:33.340585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:33.340171 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:34.356308 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.356267 5072 generic.go:356] "Generic (PLEG): container finished" podID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerID="60522e0d0c6e8b19e23b165632614349fe8b7a09a9e75c96db9388a007032c81" exitCode=0 Oct 30 06:17:34.356729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.356376 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" event={"ID":"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3","Type":"ContainerDied","Data":"60522e0d0c6e8b19e23b165632614349fe8b7a09a9e75c96db9388a007032c81"} Oct 30 06:17:34.364765 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.364721 5072 generic.go:356] "Generic (PLEG): container finished" podID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerID="87d078dd3a579c1fac2e93b53a5bdbbb5218b6d982beecf02c3f72eec7512206" exitCode=0 Oct 30 06:17:34.365962 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.365898 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" event={"ID":"43f498f0-dbf7-4beb-85d0-22978f1255a3","Type":"ContainerDied","Data":"87d078dd3a579c1fac2e93b53a5bdbbb5218b6d982beecf02c3f72eec7512206"} Oct 30 06:17:34.458093 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.458060 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:17:34.461678 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.461655 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:17:34.528811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.528336 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" podStartSLOduration=3.217881398 podStartE2EDuration="3.528312695s" totalImagesPullingTime="310.431297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:33.382496854 +0000 UTC m=+2812.986186106" watchObservedRunningTime="2026-10-30 06:17:34.528312695 +0000 UTC m=+2814.132001954" Oct 30 06:17:34.534341 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.534309 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-kube-api-access-66nhv\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-kube-api-access-66nhv\") pod \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " Oct 30 06:17:34.534585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.534383 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-envoy\") pod \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " Oct 30 06:17:34.534585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.534409 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-data\") pod \"43f498f0-dbf7-4beb-85d0-22978f1255a3\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " Oct 30 06:17:34.534585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.534435 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-credential-socket\") pod \"43f498f0-dbf7-4beb-85d0-22978f1255a3\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " Oct 30 06:17:34.534585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.534528 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-data\") pod \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " Oct 30 06:17:34.535114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.534599 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-podinfo\") pod \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " Oct 30 06:17:34.535114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.534685 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-credential-socket\") pod \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " Oct 30 06:17:34.535407 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.535377 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-credential-socket" pod "43f498f0-dbf7-4beb-85d0-22978f1255a3" (UID: "43f498f0-dbf7-4beb-85d0-22978f1255a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.535741 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.535692 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-data" pod "43f498f0-dbf7-4beb-85d0-22978f1255a3" (UID: "43f498f0-dbf7-4beb-85d0-22978f1255a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.536261 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.536232 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-credential-socket" pod "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" (UID: "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.536496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.536433 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-data" pod "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" (UID: "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.536658 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.536625 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-envoy\") pod \"43f498f0-dbf7-4beb-85d0-22978f1255a3\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " Oct 30 06:17:34.536743 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.536709 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f498f0-dbf7-4beb-85d0-22978f1255a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f498f0-dbf7-4beb-85d0-22978f1255a3-istiod-ca-cert\") pod \"43f498f0-dbf7-4beb-85d0-22978f1255a3\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " Oct 30 06:17:34.537030 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.536983 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-token\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-token\") pod \"43f498f0-dbf7-4beb-85d0-22978f1255a3\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " Oct 30 06:17:34.537114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537088 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-kube-api-access-mhrkx\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-kube-api-access-mhrkx\") pod \"43f498f0-dbf7-4beb-85d0-22978f1255a3\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " Oct 30 06:17:34.537184 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537130 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-certs\") pod \"43f498f0-dbf7-4beb-85d0-22978f1255a3\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " Oct 30 06:17:34.537244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537179 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-socket\") pod \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " Oct 30 06:17:34.537244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537202 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f498f0-dbf7-4beb-85d0-22978f1255a3-istiod-ca-cert" pod "43f498f0-dbf7-4beb-85d0-22978f1255a3" (UID: "43f498f0-dbf7-4beb-85d0-22978f1255a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:34.537244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537237 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-certs\") pod \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " Oct 30 06:17:34.537415 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537344 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istiod-ca-cert\") pod \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " Oct 30 06:17:34.537508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537436 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-token\") pod \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\" (UID: \"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3\") " Oct 30 06:17:34.537602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537512 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-socket\") pod \"43f498f0-dbf7-4beb-85d0-22978f1255a3\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " Oct 30 06:17:34.537663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537608 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-podinfo\") pod \"43f498f0-dbf7-4beb-85d0-22978f1255a3\" (UID: \"43f498f0-dbf7-4beb-85d0-22978f1255a3\") " Oct 30 06:17:34.537971 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.537939 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-envoy" pod "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" (UID: "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.538740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.538712 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.538845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.538770 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.538845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.538790 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.538845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.538806 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.538845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.538819 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.538845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.538833 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f498f0-dbf7-4beb-85d0-22978f1255a3-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.539137 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.539055 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-certs" pod "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" (UID: "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.539196 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.539138 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-kube-api-access-66nhv" pod "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" (UID: "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3"). InnerVolumeSpecName "kube-api-access-66nhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:34.539270 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.539248 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-certs" pod "43f498f0-dbf7-4beb-85d0-22978f1255a3" (UID: "43f498f0-dbf7-4beb-85d0-22978f1255a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.539520 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.539497 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-socket" pod "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" (UID: "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.540304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.540281 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-podinfo" pod "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" (UID: "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:34.540434 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.540412 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istiod-ca-cert" pod "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" (UID: "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:34.540559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.540517 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-socket" pod "43f498f0-dbf7-4beb-85d0-22978f1255a3" (UID: "43f498f0-dbf7-4beb-85d0-22978f1255a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.542301 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.542279 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-token" pod "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" (UID: "b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:34.545550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.545526 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-envoy" pod "43f498f0-dbf7-4beb-85d0-22978f1255a3" (UID: "43f498f0-dbf7-4beb-85d0-22978f1255a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:34.545717 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.545688 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-kube-api-access-mhrkx" pod "43f498f0-dbf7-4beb-85d0-22978f1255a3" (UID: "43f498f0-dbf7-4beb-85d0-22978f1255a3"). InnerVolumeSpecName "kube-api-access-mhrkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:34.546253 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.546052 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-podinfo" pod "43f498f0-dbf7-4beb-85d0-22978f1255a3" (UID: "43f498f0-dbf7-4beb-85d0-22978f1255a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:34.547546 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.547520 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-token" pod "43f498f0-dbf7-4beb-85d0-22978f1255a3" (UID: "43f498f0-dbf7-4beb-85d0-22978f1255a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640553 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640592 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640609 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrkx\" (UniqueName: \"kubernetes.io/projected/43f498f0-dbf7-4beb-85d0-22978f1255a3-kube-api-access-mhrkx\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640623 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640638 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640651 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640664 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640672 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640680 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f498f0-dbf7-4beb-85d0-22978f1255a3-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640689 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f498f0-dbf7-4beb-85d0-22978f1255a3-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640700 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66nhv\" (UniqueName: \"kubernetes.io/projected/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-kube-api-access-66nhv\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:34.640770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:34.640709 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:35.255333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.255296 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z"] Oct 30 06:17:35.255989 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.255593 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" podUID="d5d8477b-0754-4ff5-9aac-eab6828775ee" containerName="payload-pre-processing" containerID="cri-o://3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6" gracePeriod=30 Oct 30 06:17:35.392774 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.392745 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" Oct 30 06:17:35.393553 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.392770 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb" event={"ID":"43f498f0-dbf7-4beb-85d0-22978f1255a3","Type":"ContainerDied","Data":"0ddd710897437cf4fb26e8f263c578f0cf5f0346d11001d751dbf8dd3253a889"} Oct 30 06:17:35.393553 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.393408 5072 scope.go:122] "RemoveContainer" containerID="87d078dd3a579c1fac2e93b53a5bdbbb5218b6d982beecf02c3f72eec7512206" Oct 30 06:17:35.394908 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.394827 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" event={"ID":"b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3","Type":"ContainerDied","Data":"bc71d3a6835ad5b97dab6c0a3db9bf386f7fa9af6d42fa5164f0c9d6db940c41"} Oct 30 06:17:35.395082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.395064 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l" Oct 30 06:17:35.417640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.417616 5072 scope.go:122] "RemoveContainer" containerID="60522e0d0c6e8b19e23b165632614349fe8b7a09a9e75c96db9388a007032c81" Oct 30 06:17:35.468834 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.468800 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l"] Oct 30 06:17:35.476330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.476245 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-kmd5l"] Oct 30 06:17:35.506990 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.506936 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb"] Oct 30 06:17:35.520485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.516866 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-c2pnb"] Oct 30 06:17:35.954122 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:35.954092 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:17:36.027841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.027783 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d8477b-0754-4ff5-9aac-eab6828775ee-kube-api-access-gmmwk\" (UniqueName: \"kubernetes.io/projected/d5d8477b-0754-4ff5-9aac-eab6828775ee-kube-api-access-gmmwk\") pod \"d5d8477b-0754-4ff5-9aac-eab6828775ee\" (UID: \"d5d8477b-0754-4ff5-9aac-eab6828775ee\") " Oct 30 06:17:36.028496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.028470 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d8477b-0754-4ff5-9aac-eab6828775ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d8477b-0754-4ff5-9aac-eab6828775ee-plugins-config-volume\") pod \"d5d8477b-0754-4ff5-9aac-eab6828775ee\" (UID: \"d5d8477b-0754-4ff5-9aac-eab6828775ee\") " Oct 30 06:17:36.029471 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.029434 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d8477b-0754-4ff5-9aac-eab6828775ee-plugins-config-volume" pod "d5d8477b-0754-4ff5-9aac-eab6828775ee" (UID: "d5d8477b-0754-4ff5-9aac-eab6828775ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:36.033846 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.033802 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d8477b-0754-4ff5-9aac-eab6828775ee-kube-api-access-gmmwk" pod "d5d8477b-0754-4ff5-9aac-eab6828775ee" (UID: "d5d8477b-0754-4ff5-9aac-eab6828775ee"). InnerVolumeSpecName "kube-api-access-gmmwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:36.165236 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.145973 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmmwk\" (UniqueName: \"kubernetes.io/projected/d5d8477b-0754-4ff5-9aac-eab6828775ee-kube-api-access-gmmwk\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.175833 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.169615 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d8477b-0754-4ff5-9aac-eab6828775ee-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.406008 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.405974 5072 generic.go:356] "Generic (PLEG): container finished" podID="d5d8477b-0754-4ff5-9aac-eab6828775ee" containerID="3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6" exitCode=0 Oct 30 06:17:36.406540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.406033 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" Oct 30 06:17:36.406540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.406028 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" event={"ID":"d5d8477b-0754-4ff5-9aac-eab6828775ee","Type":"ContainerDied","Data":"3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6"} Oct 30 06:17:36.406540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.406108 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" event={"ID":"d5d8477b-0754-4ff5-9aac-eab6828775ee","Type":"ContainerDied","Data":"0cda668fa9901f546d9bf7dc8eff0335d6fa7d939208c9c9ff0ab891fe3aa5ce"} Oct 30 06:17:36.406540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.406159 5072 scope.go:122] "RemoveContainer" containerID="3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6" Oct 30 06:17:36.421885 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.421858 5072 scope.go:122] "RemoveContainer" containerID="3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6" Oct 30 06:17:36.422472 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:17:36.422430 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6\": container with ID starting with 3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6 not found: ID does not exist" containerID="3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6" Oct 30 06:17:36.422670 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.422623 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6"} err="failed to get container status \"3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6\": rpc error: code = NotFound desc = could not find container \"3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6\": container with ID starting with 3ee2410479f84589bae1ad2837d475c536f3677ab91e10cb0f1a3aabbfa4daa6 not found: ID does not exist" Oct 30 06:17:36.466838 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.466802 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z"] Oct 30 06:17:36.473887 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.473858 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z"] Oct 30 06:17:36.525985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.525950 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" path="/var/lib/kubelet/pods/43f498f0-dbf7-4beb-85d0-22978f1255a3/volumes" Oct 30 06:17:36.526508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.526492 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" path="/var/lib/kubelet/pods/b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3/volumes" Oct 30 06:17:36.526904 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.526890 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d8477b-0754-4ff5-9aac-eab6828775ee" path="/var/lib/kubelet/pods/d5d8477b-0754-4ff5-9aac-eab6828775ee/volumes" Oct 30 06:17:36.802341 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:36.802303 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c201cd-5c6fdd9875-pfp8z" podUID="d5d8477b-0754-4ff5-9aac-eab6828775ee" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.196:9004: i/o timeout" Oct 30 06:17:37.418762 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:37.418729 5072 generic.go:356] "Generic (PLEG): container finished" podID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerID="95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34" exitCode=0 Oct 30 06:17:37.419204 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:37.418807 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" event={"ID":"854b69be-96e5-4912-8ba9-1df7ce008e7c","Type":"ContainerDied","Data":"95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34"} Oct 30 06:17:37.419326 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:37.419304 5072 scope.go:122] "RemoveContainer" containerID="95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34" Oct 30 06:17:37.421397 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:37.421368 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:17:37.421482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:37.421442 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:17:37.421532 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:37.421518 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:17:38.444232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:38.444193 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" event={"ID":"854b69be-96e5-4912-8ba9-1df7ce008e7c","Type":"ContainerStarted","Data":"06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d"} Oct 30 06:17:38.892441 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:38.892387 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:17:38.894115 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:38.894086 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 06:17:38.894300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:38.894133 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 06:17:39.373542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:39.373511 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:17:39.892181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:39.892135 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 06:17:39.892634 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:39.892204 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 06:17:40.892748 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:40.892717 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 06:17:40.893241 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:40.892767 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 06:17:41.891871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:41.891836 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 06:17:41.892066 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:41.891885 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 06:17:42.770338 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:42.770292 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:17:42.896414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:42.892710 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 06:17:42.896414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:42.892777 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 06:17:42.903172 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:42.903049 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m"] Oct 30 06:17:42.904558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:42.903314 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" containerID="cri-o://1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d" gracePeriod=30 Oct 30 06:17:43.891877 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:43.891842 5072 patch_prober.go:29] interesting pod/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 30 06:17:43.892417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:43.891895 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 30 06:17:44.897403 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:44.897369 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:17:44.898735 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:44.898711 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:17:48.275993 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.275969 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:17:48.424589 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.424551 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b389edd-730e-400c-9127-8da67568ccf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b389edd-730e-400c-9127-8da67568ccf5-istio-podinfo\") pod \"9b389edd-730e-400c-9127-8da67568ccf5\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " Oct 30 06:17:48.424800 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.424634 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-envoy\") pod \"9b389edd-730e-400c-9127-8da67568ccf5\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " Oct 30 06:17:48.424800 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.424664 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-certs\") pod \"9b389edd-730e-400c-9127-8da67568ccf5\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " Oct 30 06:17:48.424800 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.424741 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-socket\") pod \"9b389edd-730e-400c-9127-8da67568ccf5\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " Oct 30 06:17:48.424974 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.424795 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-kube-api-access-2qtwr\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-kube-api-access-2qtwr\") pod \"9b389edd-730e-400c-9127-8da67568ccf5\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " Oct 30 06:17:48.425104 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.425048 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-certs" pod "9b389edd-730e-400c-9127-8da67568ccf5" (UID: "9b389edd-730e-400c-9127-8da67568ccf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.425104 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.425067 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-socket" pod "9b389edd-730e-400c-9127-8da67568ccf5" (UID: "9b389edd-730e-400c-9127-8da67568ccf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.425320 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.425245 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-credential-socket" pod "9b389edd-730e-400c-9127-8da67568ccf5" (UID: "9b389edd-730e-400c-9127-8da67568ccf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.425431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.425406 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-credential-socket\") pod \"9b389edd-730e-400c-9127-8da67568ccf5\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " Oct 30 06:17:48.425591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.425574 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-data\") pod \"9b389edd-730e-400c-9127-8da67568ccf5\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " Oct 30 06:17:48.425695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.425681 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b389edd-730e-400c-9127-8da67568ccf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b389edd-730e-400c-9127-8da67568ccf5-istiod-ca-cert\") pod \"9b389edd-730e-400c-9127-8da67568ccf5\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " Oct 30 06:17:48.425864 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.425848 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-istio-token\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-istio-token\") pod \"9b389edd-730e-400c-9127-8da67568ccf5\" (UID: \"9b389edd-730e-400c-9127-8da67568ccf5\") " Oct 30 06:17:48.426396 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.425850 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-data" pod "9b389edd-730e-400c-9127-8da67568ccf5" (UID: "9b389edd-730e-400c-9127-8da67568ccf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.426695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.426126 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b389edd-730e-400c-9127-8da67568ccf5-istiod-ca-cert" pod "9b389edd-730e-400c-9127-8da67568ccf5" (UID: "9b389edd-730e-400c-9127-8da67568ccf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:48.426695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.426432 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.426695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.426651 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.426695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.426672 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.427523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.427498 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-envoy" pod "9b389edd-730e-400c-9127-8da67568ccf5" (UID: "9b389edd-730e-400c-9127-8da67568ccf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:48.427672 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.427647 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b389edd-730e-400c-9127-8da67568ccf5-istio-podinfo" pod "9b389edd-730e-400c-9127-8da67568ccf5" (UID: "9b389edd-730e-400c-9127-8da67568ccf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:48.427819 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.427797 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-kube-api-access-2qtwr" pod "9b389edd-730e-400c-9127-8da67568ccf5" (UID: "9b389edd-730e-400c-9127-8da67568ccf5"). InnerVolumeSpecName "kube-api-access-2qtwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:48.428276 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.428259 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-istio-token" pod "9b389edd-730e-400c-9127-8da67568ccf5" (UID: "9b389edd-730e-400c-9127-8da67568ccf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:48.528163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.528076 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qtwr\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-kube-api-access-2qtwr\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.528163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.528113 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.528163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.528127 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b389edd-730e-400c-9127-8da67568ccf5-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.528163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.528139 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b389edd-730e-400c-9127-8da67568ccf5-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.528163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.528164 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b389edd-730e-400c-9127-8da67568ccf5-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.528443 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.528177 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b389edd-730e-400c-9127-8da67568ccf5-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:48.546479 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.546433 5072 generic.go:356] "Generic (PLEG): container finished" podID="9b389edd-730e-400c-9127-8da67568ccf5" containerID="1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d" exitCode=0 Oct 30 06:17:48.546664 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.546507 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" event={"ID":"9b389edd-730e-400c-9127-8da67568ccf5","Type":"ContainerDied","Data":"1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d"} Oct 30 06:17:48.546664 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.546525 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" Oct 30 06:17:48.546664 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.546564 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m" event={"ID":"9b389edd-730e-400c-9127-8da67568ccf5","Type":"ContainerDied","Data":"c811065776e7d7a414ea4193e590cc33073859b6ef745d9cb3046077a80deb94"} Oct 30 06:17:48.546664 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.546588 5072 scope.go:122] "RemoveContainer" containerID="1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d" Oct 30 06:17:48.558998 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.558977 5072 scope.go:122] "RemoveContainer" containerID="41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1" Oct 30 06:17:48.569115 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.569092 5072 scope.go:122] "RemoveContainer" containerID="1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d" Oct 30 06:17:48.569425 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:17:48.569401 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d\": container with ID starting with 1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d not found: ID does not exist" containerID="1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d" Oct 30 06:17:48.569583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.569434 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d"} err="failed to get container status \"1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d\": rpc error: code = NotFound desc = could not find container \"1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d\": container with ID starting with 1aa2b96e013520d23dd871b5dc7222ca0a8ad1072721f8fa8e1d06e6f3af9a7d not found: ID does not exist" Oct 30 06:17:48.569583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.569452 5072 scope.go:122] "RemoveContainer" containerID="41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1" Oct 30 06:17:48.569730 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:17:48.569713 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1\": container with ID starting with 41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1 not found: ID does not exist" containerID="41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1" Oct 30 06:17:48.569775 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.569738 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1"} err="failed to get container status \"41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1\": rpc error: code = NotFound desc = could not find container \"41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1\": container with ID starting with 41df1000c21cba1563381a4605231f7fcf54b7142d9859926247c11dd28ca8d1 not found: ID does not exist" Oct 30 06:17:48.596831 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.596794 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m"] Oct 30 06:17:48.611584 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:48.611554 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c201cd-openshift-default-b7586fbcc-4qn8m"] Oct 30 06:17:50.524452 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:50.524366 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b389edd-730e-400c-9127-8da67568ccf5" path="/var/lib/kubelet/pods/9b389edd-730e-400c-9127-8da67568ccf5/volumes" Oct 30 06:17:56.565403 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.565368 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr"] Oct 30 06:17:56.565915 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.565597 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" podUID="e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" containerName="maas-api" containerID="cri-o://16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8" gracePeriod=30 Oct 30 06:17:56.913981 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.913949 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:17:56.976593 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.976137 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-kube-api-access-4xjsn\" (UniqueName: \"kubernetes.io/projected/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-kube-api-access-4xjsn\") pod \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " Oct 30 06:17:56.976822 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.976653 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-maas-api-tls\") pod \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " Oct 30 06:17:56.976822 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.976751 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-openshift-service-ca\") pod \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " Oct 30 06:17:56.976822 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.976815 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-metrics-certs\") pod \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\" (UID: \"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386\") " Oct 30 06:17:56.977521 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.977487 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-openshift-service-ca" pod "e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" (UID: "e9a2f6c9-1a4c-4d3c-98fd-e39e30425386"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:56.977943 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.977920 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:56.978867 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.978846 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-kube-api-access-4xjsn" pod "e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" (UID: "e9a2f6c9-1a4c-4d3c-98fd-e39e30425386"). InnerVolumeSpecName "kube-api-access-4xjsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:56.988421 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.981130 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-maas-api-tls" pod "e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" (UID: "e9a2f6c9-1a4c-4d3c-98fd-e39e30425386"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:56.988421 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:56.981164 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-metrics-certs" pod "e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" (UID: "e9a2f6c9-1a4c-4d3c-98fd-e39e30425386"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:57.080293 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.080102 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xjsn\" (UniqueName: \"kubernetes.io/projected/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-kube-api-access-4xjsn\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:57.080663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.080578 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:57.080663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.080607 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:17:57.623358 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.623323 5072 generic.go:356] "Generic (PLEG): container finished" podID="e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" containerID="16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8" exitCode=0 Oct 30 06:17:57.623358 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.623363 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" event={"ID":"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386","Type":"ContainerDied","Data":"16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8"} Oct 30 06:17:57.623963 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.623388 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" Oct 30 06:17:57.623963 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.623404 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr" event={"ID":"e9a2f6c9-1a4c-4d3c-98fd-e39e30425386","Type":"ContainerDied","Data":"366d2f612d2342b206924aaa60b2170a8a74089d06ba04cea39202a79186d6e4"} Oct 30 06:17:57.623963 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.623424 5072 scope.go:122] "RemoveContainer" containerID="16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8" Oct 30 06:17:57.637093 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.637068 5072 scope.go:122] "RemoveContainer" containerID="16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8" Oct 30 06:17:57.637403 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:17:57.637378 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8\": container with ID starting with 16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8 not found: ID does not exist" containerID="16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8" Oct 30 06:17:57.637505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.637416 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8"} err="failed to get container status \"16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8\": rpc error: code = NotFound desc = could not find container \"16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8\": container with ID starting with 16df2d980c88b5633a2ba4b3626dcf55ef56a272d2356ae93e0c96380bf6e7b8 not found: ID does not exist" Oct 30 06:17:57.673289 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.673225 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr"] Oct 30 06:17:57.681046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:57.681017 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-370dfa-644d9b67c7-ghvgr"] Oct 30 06:17:58.523583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:17:58.523537 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" path="/var/lib/kubelet/pods/e9a2f6c9-1a4c-4d3c-98fd-e39e30425386/volumes" Oct 30 06:18:00.767687 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:18:00.767640 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:18:12.334752 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.334708 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl"] Oct 30 06:18:12.335489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335471 5072 cpu_manager.go:414] "containerMap: removing container" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" Oct 30 06:18:12.335489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335488 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335516 5072 cpu_manager.go:414] "containerMap: removing container" podUID="e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" containerName="maas-api" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335521 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" containerName="maas-api" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335532 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d5d8477b-0754-4ff5-9aac-eab6828775ee" containerName="payload-pre-processing" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335537 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d8477b-0754-4ff5-9aac-eab6828775ee" containerName="payload-pre-processing" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335547 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335552 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335574 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335580 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335597 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335602 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335724 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" Oct 30 06:18:12.335720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335737 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f498f0-dbf7-4beb-85d0-22978f1255a3" containerName="istio-proxy" Oct 30 06:18:12.336394 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335762 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a2f6c9-1a4c-4d3c-98fd-e39e30425386" containerName="maas-api" Oct 30 06:18:12.336394 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335771 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d8477b-0754-4ff5-9aac-eab6828775ee" containerName="payload-pre-processing" Oct 30 06:18:12.336394 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.335788 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bbf04c-8a20-4c7e-a61e-94e9db4b2ed3" containerName="istio-proxy" Oct 30 06:18:12.339898 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.339876 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.367974 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.367942 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7fdb95-serving-cert\"" Oct 30 06:18:12.371573 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.371543 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl"] Oct 30 06:18:12.474903 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.474866 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgmx\" (UniqueName: \"kubernetes.io/projected/9a235824-ceb8-4424-9404-64c98bd44b39-kube-api-access-gsgmx\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.475071 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.475022 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-metrics-certs\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.475071 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.475051 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a235824-ceb8-4424-9404-64c98bd44b39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.475207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.475187 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.578417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.578378 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-metrics-certs\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.578684 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.578449 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a235824-ceb8-4424-9404-64c98bd44b39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.578684 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.578553 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.578828 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.578753 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgmx\" (UniqueName: \"kubernetes.io/projected/9a235824-ceb8-4424-9404-64c98bd44b39-kube-api-access-gsgmx\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.582735 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.580347 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a235824-ceb8-4424-9404-64c98bd44b39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.589989 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.589911 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-metrics-certs\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.592430 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.592401 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.624558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.624521 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgmx\" (UniqueName: \"kubernetes.io/projected/9a235824-ceb8-4424-9404-64c98bd44b39-kube-api-access-gsgmx\") pod \"maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.656296 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.656264 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:12.971293 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:12.971260 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl"] Oct 30 06:18:12.971842 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:18:12.971813 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a235824_ceb8_4424_9404_64c98bd44b39.slice/crio-99a4aa5269498e0dad09ec8d4f6e664896a92d23d7c4ad7eaa7bdc4ce7d3d5a5 WatchSource:0}: Error finding container 99a4aa5269498e0dad09ec8d4f6e664896a92d23d7c4ad7eaa7bdc4ce7d3d5a5: Status 404 returned error can't find the container with id 99a4aa5269498e0dad09ec8d4f6e664896a92d23d7c4ad7eaa7bdc4ce7d3d5a5 Oct 30 06:18:13.816908 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:13.816522 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" event={"ID":"9a235824-ceb8-4424-9404-64c98bd44b39","Type":"ContainerStarted","Data":"d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4"} Oct 30 06:18:13.817301 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:13.816932 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" event={"ID":"9a235824-ceb8-4424-9404-64c98bd44b39","Type":"ContainerStarted","Data":"99a4aa5269498e0dad09ec8d4f6e664896a92d23d7c4ad7eaa7bdc4ce7d3d5a5"} Oct 30 06:18:13.817301 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:13.816969 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:13.859675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:13.859623 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" podStartSLOduration=1.453310159 podStartE2EDuration="1.859608705s" totalImagesPullingTime="406.298546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:13.855574277 +0000 UTC m=+2853.459263548" watchObservedRunningTime="2026-10-30 06:18:13.859608705 +0000 UTC m=+2853.463297962" Oct 30 06:18:15.360998 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:15.360964 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25"] Oct 30 06:18:15.986834 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:15.986801 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr"] Oct 30 06:18:15.987889 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:15.987868 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b389edd-730e-400c-9127-8da67568ccf5" containerName="istio-proxy" Oct 30 06:18:15.992150 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:15.992114 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:16.027328 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.027292 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr"] Oct 30 06:18:16.169494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.149049 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fb9345-dockercfg-h6ks4\"" Oct 30 06:18:16.169494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.149205 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fb9345\"" Oct 30 06:18:16.254022 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.253860 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d2ac8a-7330-4daa-88f3-2e6db8313230-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr\" (UID: \"73d2ac8a-7330-4daa-88f3-2e6db8313230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:16.254377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.254353 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngzh\" (UniqueName: \"kubernetes.io/projected/73d2ac8a-7330-4daa-88f3-2e6db8313230-kube-api-access-vngzh\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr\" (UID: \"73d2ac8a-7330-4daa-88f3-2e6db8313230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:16.356705 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.356655 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d2ac8a-7330-4daa-88f3-2e6db8313230-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr\" (UID: \"73d2ac8a-7330-4daa-88f3-2e6db8313230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:16.356890 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.356826 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngzh\" (UniqueName: \"kubernetes.io/projected/73d2ac8a-7330-4daa-88f3-2e6db8313230-kube-api-access-vngzh\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr\" (UID: \"73d2ac8a-7330-4daa-88f3-2e6db8313230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:16.357587 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.357567 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d2ac8a-7330-4daa-88f3-2e6db8313230-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr\" (UID: \"73d2ac8a-7330-4daa-88f3-2e6db8313230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:16.381449 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.381418 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngzh\" (UniqueName: \"kubernetes.io/projected/73d2ac8a-7330-4daa-88f3-2e6db8313230-kube-api-access-vngzh\") pod \"payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr\" (UID: \"73d2ac8a-7330-4daa-88f3-2e6db8313230\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:16.623067 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.623029 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:16.932104 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:16.932068 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr"] Oct 30 06:18:16.933554 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:18:16.933521 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d2ac8a_7330_4daa_88f3_2e6db8313230.slice/crio-b1308df0d8808972bc060c9b31289e9c2f5e7ef80b53db167fe4b089ae9b4dc3 WatchSource:0}: Error finding container b1308df0d8808972bc060c9b31289e9c2f5e7ef80b53db167fe4b089ae9b4dc3: Status 404 returned error can't find the container with id b1308df0d8808972bc060c9b31289e9c2f5e7ef80b53db167fe4b089ae9b4dc3 Oct 30 06:18:17.886480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:17.883139 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" event={"ID":"73d2ac8a-7330-4daa-88f3-2e6db8313230","Type":"ContainerStarted","Data":"817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44"} Oct 30 06:18:17.886480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:17.883214 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" event={"ID":"73d2ac8a-7330-4daa-88f3-2e6db8313230","Type":"ContainerStarted","Data":"b1308df0d8808972bc060c9b31289e9c2f5e7ef80b53db167fe4b089ae9b4dc3"} Oct 30 06:18:17.886480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:17.884597 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:17.933529 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:17.933444 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" podStartSLOduration=2.9334301910000002 podStartE2EDuration="2.933430191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:17.931021591 +0000 UTC m=+2857.534710849" watchObservedRunningTime="2026-10-30 06:18:17.933430191 +0000 UTC m=+2857.537119447" Oct 30 06:18:19.829424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:19.829389 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:18:20.361805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.361755 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" podUID="08171d10-1ddb-4879-bfb6-851928e94d79" containerName="payload-pre-processing" containerID="cri-o://6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce" gracePeriod=25 Oct 30 06:18:20.669097 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.669068 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 06:18:20.753513 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.753482 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08171d10-1ddb-4879-bfb6-851928e94d79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08171d10-1ddb-4879-bfb6-851928e94d79-plugins-config-volume\") pod \"08171d10-1ddb-4879-bfb6-851928e94d79\" (UID: \"08171d10-1ddb-4879-bfb6-851928e94d79\") " Oct 30 06:18:20.753764 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.753749 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08171d10-1ddb-4879-bfb6-851928e94d79-kube-api-access-h6pt2\" (UniqueName: \"kubernetes.io/projected/08171d10-1ddb-4879-bfb6-851928e94d79-kube-api-access-h6pt2\") pod \"08171d10-1ddb-4879-bfb6-851928e94d79\" (UID: \"08171d10-1ddb-4879-bfb6-851928e94d79\") " Oct 30 06:18:20.753955 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.753928 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08171d10-1ddb-4879-bfb6-851928e94d79-plugins-config-volume" pod "08171d10-1ddb-4879-bfb6-851928e94d79" (UID: "08171d10-1ddb-4879-bfb6-851928e94d79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:20.754395 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.754374 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08171d10-1ddb-4879-bfb6-851928e94d79-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:20.756137 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.756102 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08171d10-1ddb-4879-bfb6-851928e94d79-kube-api-access-h6pt2" pod "08171d10-1ddb-4879-bfb6-851928e94d79" (UID: "08171d10-1ddb-4879-bfb6-851928e94d79"). InnerVolumeSpecName "kube-api-access-h6pt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:20.855922 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.855888 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6pt2\" (UniqueName: \"kubernetes.io/projected/08171d10-1ddb-4879-bfb6-851928e94d79-kube-api-access-h6pt2\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:20.913891 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.913809 5072 generic.go:356] "Generic (PLEG): container finished" podID="08171d10-1ddb-4879-bfb6-851928e94d79" containerID="6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce" exitCode=0 Oct 30 06:18:20.913891 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.913866 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" event={"ID":"08171d10-1ddb-4879-bfb6-851928e94d79","Type":"ContainerDied","Data":"6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce"} Oct 30 06:18:20.913891 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.913876 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" Oct 30 06:18:20.914148 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.913907 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25" event={"ID":"08171d10-1ddb-4879-bfb6-851928e94d79","Type":"ContainerDied","Data":"b75f921670d09b6a248421ffb613b30610cb699b5af0dd8a842913cec436728e"} Oct 30 06:18:20.914148 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.913928 5072 scope.go:122] "RemoveContainer" containerID="6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce" Oct 30 06:18:20.925789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.925406 5072 scope.go:122] "RemoveContainer" containerID="6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce" Oct 30 06:18:20.925789 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:18:20.925767 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce\": container with ID starting with 6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce not found: ID does not exist" containerID="6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce" Oct 30 06:18:20.925892 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.925798 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce"} err="failed to get container status \"6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce\": rpc error: code = NotFound desc = could not find container \"6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce\": container with ID starting with 6d2e6cce23f6df8803e0b8b89b9106e885ebce90d207690d112f2ee253f622ce not found: ID does not exist" Oct 30 06:18:20.961745 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.961710 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25"] Oct 30 06:18:20.968159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:20.968119 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d96a1f-7884d8d54c-48w25"] Oct 30 06:18:22.519399 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:22.519364 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08171d10-1ddb-4879-bfb6-851928e94d79" path="/var/lib/kubelet/pods/08171d10-1ddb-4879-bfb6-851928e94d79/volumes" Oct 30 06:18:29.906079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:29.906045 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:18:30.713632 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:18:30.713592 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:18:33.988449 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:33.988408 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2"] Oct 30 06:18:33.989731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:33.989705 5072 cpu_manager.go:414] "containerMap: removing container" podUID="08171d10-1ddb-4879-bfb6-851928e94d79" containerName="payload-pre-processing" Oct 30 06:18:33.989731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:33.989729 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08171d10-1ddb-4879-bfb6-851928e94d79" containerName="payload-pre-processing" Oct 30 06:18:33.989947 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:33.989929 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="08171d10-1ddb-4879-bfb6-851928e94d79" containerName="payload-pre-processing" Oct 30 06:18:33.995344 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:33.995318 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.024400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.022689 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-certs\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.024400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.022853 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istiod-ca-cert\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.024400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.022887 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-envoy\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.024400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.022945 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-credential-socket\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.024400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.022974 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-data\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.024400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.023000 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflvp\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-kube-api-access-bflvp\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.024400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.023096 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-socket\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.024400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.023122 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-token\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.024400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.023237 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-podinfo\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.029496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.025489 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2"] Oct 30 06:18:34.029496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.027551 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a8b892-openshift-default-dockercfg-vcp59\"" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.128711 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istiod-ca-cert\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.128775 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-envoy\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.128843 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-credential-socket\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.128892 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-data\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.128923 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflvp\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-kube-api-access-bflvp\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.129056 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-socket\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.129098 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-token\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.129227 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-podinfo\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.129323 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-certs\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.130494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.129911 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-certs\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.142055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.139808 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-credential-socket\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.142055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.140061 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-data\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.142055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.140709 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-socket\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.202483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.186222 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflvp\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-kube-api-access-bflvp\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.204821 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.204784 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-envoy\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.206434 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.206404 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-podinfo\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.217634 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.217600 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istiod-ca-cert\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.235483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.232251 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-token\") pod \"e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.336536 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.332844 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:34.799206 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:18:34.799173 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d97d10_ff5d_499a_a994_2721e2e3c8fa.slice/crio-e2ce6184eb2400c9932fb4f24d6bebb28e725bc7d2d702260903c4a0341efc3b WatchSource:0}: Error finding container e2ce6184eb2400c9932fb4f24d6bebb28e725bc7d2d702260903c4a0341efc3b: Status 404 returned error can't find the container with id e2ce6184eb2400c9932fb4f24d6bebb28e725bc7d2d702260903c4a0341efc3b Oct 30 06:18:34.810345 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.801960 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2"] Oct 30 06:18:34.815423 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.814056 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:18:34.815423 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.814157 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:18:34.815423 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:34.814201 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:18:35.052118 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:35.051987 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" event={"ID":"29d97d10-ff5d-499a-a994-2721e2e3c8fa","Type":"ContainerStarted","Data":"97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e"} Oct 30 06:18:35.052118 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:35.052055 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" event={"ID":"29d97d10-ff5d-499a-a994-2721e2e3c8fa","Type":"ContainerStarted","Data":"e2ce6184eb2400c9932fb4f24d6bebb28e725bc7d2d702260903c4a0341efc3b"} Oct 30 06:18:35.099645 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:35.099597 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" podStartSLOduration=2.099583693 podStartE2EDuration="2.099583693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:35.09637936 +0000 UTC m=+2874.700068610" watchObservedRunningTime="2026-10-30 06:18:35.099583693 +0000 UTC m=+2874.703272950" Oct 30 06:18:35.334905 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:35.334791 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:35.336727 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:35.336696 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 06:18:35.336903 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:35.336758 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 06:18:35.504746 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:35.504708 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf"] Oct 30 06:18:35.505007 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:35.504964 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" podUID="38da4d60-a766-494c-a6f0-91cc3577464c" containerName="maas-api" containerID="cri-o://e35222d0a1c183ce5734ae22ed399a74d59e73dec0c13a49725ac5949f87fb2e" gracePeriod=30 Oct 30 06:18:36.152482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.145264 5072 generic.go:356] "Generic (PLEG): container finished" podID="38da4d60-a766-494c-a6f0-91cc3577464c" containerID="e35222d0a1c183ce5734ae22ed399a74d59e73dec0c13a49725ac5949f87fb2e" exitCode=0 Oct 30 06:18:36.197545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.145325 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" event={"ID":"38da4d60-a766-494c-a6f0-91cc3577464c","Type":"ContainerDied","Data":"e35222d0a1c183ce5734ae22ed399a74d59e73dec0c13a49725ac5949f87fb2e"} Oct 30 06:18:36.197545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.197425 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" event={"ID":"38da4d60-a766-494c-a6f0-91cc3577464c","Type":"ContainerDied","Data":"82225f493f5a93c99d47a0f7a742100d511f1c1bbc875f885fbaf2918549c72d"} Oct 30 06:18:36.201493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.196347 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 06:18:36.201493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.200566 5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82225f493f5a93c99d47a0f7a742100d511f1c1bbc875f885fbaf2918549c72d" Oct 30 06:18:36.301901 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.301810 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-metrics-certs\") pod \"38da4d60-a766-494c-a6f0-91cc3577464c\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " Oct 30 06:18:36.301901 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.301876 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38da4d60-a766-494c-a6f0-91cc3577464c-kube-api-access-dxhdm\" (UniqueName: \"kubernetes.io/projected/38da4d60-a766-494c-a6f0-91cc3577464c-kube-api-access-dxhdm\") pod \"38da4d60-a766-494c-a6f0-91cc3577464c\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " Oct 30 06:18:36.302193 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.301970 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-maas-api-tls\") pod \"38da4d60-a766-494c-a6f0-91cc3577464c\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " Oct 30 06:18:36.302193 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.302142 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38da4d60-a766-494c-a6f0-91cc3577464c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38da4d60-a766-494c-a6f0-91cc3577464c-openshift-service-ca\") pod \"38da4d60-a766-494c-a6f0-91cc3577464c\" (UID: \"38da4d60-a766-494c-a6f0-91cc3577464c\") " Oct 30 06:18:36.328157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.328091 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38da4d60-a766-494c-a6f0-91cc3577464c-kube-api-access-dxhdm" pod "38da4d60-a766-494c-a6f0-91cc3577464c" (UID: "38da4d60-a766-494c-a6f0-91cc3577464c"). InnerVolumeSpecName "kube-api-access-dxhdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:36.337861 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.336644 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38da4d60-a766-494c-a6f0-91cc3577464c-openshift-service-ca" pod "38da4d60-a766-494c-a6f0-91cc3577464c" (UID: "38da4d60-a766-494c-a6f0-91cc3577464c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:36.341955 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.340682 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-metrics-certs" pod "38da4d60-a766-494c-a6f0-91cc3577464c" (UID: "38da4d60-a766-494c-a6f0-91cc3577464c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:18:36.440482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.423603 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-maas-api-tls" pod "38da4d60-a766-494c-a6f0-91cc3577464c" (UID: "38da4d60-a766-494c-a6f0-91cc3577464c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:18:36.514531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.509776 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:36.514531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.509812 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38da4d60-a766-494c-a6f0-91cc3577464c-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:36.514531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.509827 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38da4d60-a766-494c-a6f0-91cc3577464c-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:36.514531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.509839 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxhdm\" (UniqueName: \"kubernetes.io/projected/38da4d60-a766-494c-a6f0-91cc3577464c-kube-api-access-dxhdm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:36.629905 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:36.629883 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:37.204675 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:37.204637 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf" Oct 30 06:18:37.205137 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:37.204921 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:37.206798 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:37.206776 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:18:37.296805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:37.296771 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf"] Oct 30 06:18:37.301587 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:37.301556 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d96a1f-8486fddb5d-z47wf"] Oct 30 06:18:38.522319 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:38.522286 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38da4d60-a766-494c-a6f0-91cc3577464c" path="/var/lib/kubelet/pods/38da4d60-a766-494c-a6f0-91cc3577464c/volumes" Oct 30 06:18:43.515157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:43.515116 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m"] Oct 30 06:18:43.515793 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:43.515342 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" containerID="cri-o://76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a" gracePeriod=30 Oct 30 06:18:48.918868 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:48.918826 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 06:18:49.055109 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055072 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-credential-socket\") pod \"9b4a3b88-8b55-4812-b640-df363c7e89d4\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " Oct 30 06:18:49.055307 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055161 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4a3b88-8b55-4812-b640-df363c7e89d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4a3b88-8b55-4812-b640-df363c7e89d4-istiod-ca-cert\") pod \"9b4a3b88-8b55-4812-b640-df363c7e89d4\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " Oct 30 06:18:49.055307 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055260 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-token\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-token\") pod \"9b4a3b88-8b55-4812-b640-df363c7e89d4\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " Oct 30 06:18:49.055451 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055313 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-socket\") pod \"9b4a3b88-8b55-4812-b640-df363c7e89d4\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " Oct 30 06:18:49.055451 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055341 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-certs\") pod \"9b4a3b88-8b55-4812-b640-df363c7e89d4\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " Oct 30 06:18:49.055451 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055370 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-credential-socket" pod "9b4a3b88-8b55-4812-b640-df363c7e89d4" (UID: "9b4a3b88-8b55-4812-b640-df363c7e89d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:49.055451 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055391 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-data\") pod \"9b4a3b88-8b55-4812-b640-df363c7e89d4\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " Oct 30 06:18:49.055451 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055422 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-podinfo\") pod \"9b4a3b88-8b55-4812-b640-df363c7e89d4\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " Oct 30 06:18:49.055754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055635 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-kube-api-access-nrl67\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-kube-api-access-nrl67\") pod \"9b4a3b88-8b55-4812-b640-df363c7e89d4\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " Oct 30 06:18:49.055754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055637 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-socket" pod "9b4a3b88-8b55-4812-b640-df363c7e89d4" (UID: "9b4a3b88-8b55-4812-b640-df363c7e89d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:49.055754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055648 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-certs" pod "9b4a3b88-8b55-4812-b640-df363c7e89d4" (UID: "9b4a3b88-8b55-4812-b640-df363c7e89d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:49.055754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055718 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-envoy\") pod \"9b4a3b88-8b55-4812-b640-df363c7e89d4\" (UID: \"9b4a3b88-8b55-4812-b640-df363c7e89d4\") " Oct 30 06:18:49.055972 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.055867 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-data" pod "9b4a3b88-8b55-4812-b640-df363c7e89d4" (UID: "9b4a3b88-8b55-4812-b640-df363c7e89d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:49.056329 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.056301 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4a3b88-8b55-4812-b640-df363c7e89d4-istiod-ca-cert" pod "9b4a3b88-8b55-4812-b640-df363c7e89d4" (UID: "9b4a3b88-8b55-4812-b640-df363c7e89d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:49.056521 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.056503 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.056596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.056527 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.056596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.056542 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.056596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.056555 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.056596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.056569 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4a3b88-8b55-4812-b640-df363c7e89d4-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.059015 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.058977 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-kube-api-access-nrl67" pod "9b4a3b88-8b55-4812-b640-df363c7e89d4" (UID: "9b4a3b88-8b55-4812-b640-df363c7e89d4"). InnerVolumeSpecName "kube-api-access-nrl67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:49.059162 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.059006 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-podinfo" pod "9b4a3b88-8b55-4812-b640-df363c7e89d4" (UID: "9b4a3b88-8b55-4812-b640-df363c7e89d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:18:49.059593 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.059570 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-envoy" pod "9b4a3b88-8b55-4812-b640-df363c7e89d4" (UID: "9b4a3b88-8b55-4812-b640-df363c7e89d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:49.060203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.060176 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-token" pod "9b4a3b88-8b55-4812-b640-df363c7e89d4" (UID: "9b4a3b88-8b55-4812-b640-df363c7e89d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:49.157997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.157873 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.157997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.157930 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrl67\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-kube-api-access-nrl67\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.157997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.157942 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.157997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.157950 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4a3b88-8b55-4812-b640-df363c7e89d4-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.497517 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.497386 5072 generic.go:356] "Generic (PLEG): container finished" podID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerID="76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a" exitCode=0 Oct 30 06:18:49.497517 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.497443 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" event={"ID":"9b4a3b88-8b55-4812-b640-df363c7e89d4","Type":"ContainerDied","Data":"76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a"} Oct 30 06:18:49.497517 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.497450 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" Oct 30 06:18:49.497517 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.497504 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m" event={"ID":"9b4a3b88-8b55-4812-b640-df363c7e89d4","Type":"ContainerDied","Data":"d81a2d976acadb8149eef64a8f8d6b56b3267c4c77c0bfb59fd443d0b385986a"} Oct 30 06:18:49.497857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.497526 5072 scope.go:122] "RemoveContainer" containerID="76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a" Oct 30 06:18:49.509898 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.509875 5072 scope.go:122] "RemoveContainer" containerID="5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd" Oct 30 06:18:49.522654 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.522631 5072 scope.go:122] "RemoveContainer" containerID="76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a" Oct 30 06:18:49.523013 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:18:49.522983 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a\": container with ID starting with 76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a not found: ID does not exist" containerID="76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a" Oct 30 06:18:49.523154 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.523017 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a"} err="failed to get container status \"76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a\": rpc error: code = NotFound desc = could not find container \"76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a\": container with ID starting with 76111da412e613e6cb07b8bbe58395acec29c768ec9211c7d9c649c1129c0a2a not found: ID does not exist" Oct 30 06:18:49.523154 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.523034 5072 scope.go:122] "RemoveContainer" containerID="5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd" Oct 30 06:18:49.523317 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:18:49.523290 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd\": container with ID starting with 5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd not found: ID does not exist" containerID="5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd" Oct 30 06:18:49.523392 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.523325 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd"} err="failed to get container status \"5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd\": rpc error: code = NotFound desc = could not find container \"5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd\": container with ID starting with 5454d4b128f8039a2313d510a60fd30ed91035eeaed08336f8655d02e27039cd not found: ID does not exist" Oct 30 06:18:49.560117 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.560086 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m"] Oct 30 06:18:49.569332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:49.569306 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d96a1f-openshift-default-75d7bf6889-jsm6m"] Oct 30 06:18:50.564483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:50.557105 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" path="/var/lib/kubelet/pods/9b4a3b88-8b55-4812-b640-df363c7e89d4/volumes" Oct 30 06:18:50.603181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:18:50.603149 5072 scope.go:122] "RemoveContainer" containerID="e35222d0a1c183ce5734ae22ed399a74d59e73dec0c13a49725ac5949f87fb2e" Oct 30 06:19:00.712542 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:19:00.712493 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:19:08.324484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.323682 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8"] Oct 30 06:19:08.325051 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.324874 5072 cpu_manager.go:414] "containerMap: removing container" podUID="38da4d60-a766-494c-a6f0-91cc3577464c" containerName="maas-api" Oct 30 06:19:08.325051 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.324897 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38da4d60-a766-494c-a6f0-91cc3577464c" containerName="maas-api" Oct 30 06:19:08.325051 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.324990 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" Oct 30 06:19:08.325051 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.324999 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" Oct 30 06:19:08.325051 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.325026 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" Oct 30 06:19:08.325051 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.325033 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" Oct 30 06:19:08.325472 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.325191 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" Oct 30 06:19:08.325472 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.325208 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4a3b88-8b55-4812-b640-df363c7e89d4" containerName="istio-proxy" Oct 30 06:19:08.325472 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.325220 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="38da4d60-a766-494c-a6f0-91cc3577464c" containerName="maas-api" Oct 30 06:19:08.363486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.359748 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:08.376483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.374707 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8"] Oct 30 06:19:08.419485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.409214 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-07b278-dockercfg-cjm6p\"" Oct 30 06:19:08.419485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.409384 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-07b278\"" Oct 30 06:19:08.501528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.494757 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz7n8\" (UniqueName: \"kubernetes.io/projected/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-kube-api-access-dz7n8\") pod \"payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8\" (UID: \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:08.501528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.494818 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8\" (UID: \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:08.596051 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.595963 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz7n8\" (UniqueName: \"kubernetes.io/projected/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-kube-api-access-dz7n8\") pod \"payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8\" (UID: \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:08.596051 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.596005 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8\" (UID: \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:08.596808 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.596782 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8\" (UID: \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:08.619377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.619344 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz7n8\" (UniqueName: \"kubernetes.io/projected/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-kube-api-access-dz7n8\") pod \"payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8\" (UID: \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:08.884230 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:08.884195 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:09.276985 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:19:09.276951 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ba9fdd_9ed9_488c_ae0b_5d56573bfd46.slice/crio-c919801cf527a5a97410e5ea5fdda3456b5c3d62f2e67d48d84e74bfe3c8f9c1 WatchSource:0}: Error finding container c919801cf527a5a97410e5ea5fdda3456b5c3d62f2e67d48d84e74bfe3c8f9c1: Status 404 returned error can't find the container with id c919801cf527a5a97410e5ea5fdda3456b5c3d62f2e67d48d84e74bfe3c8f9c1 Oct 30 06:19:09.277223 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:09.277198 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8"] Oct 30 06:19:09.689815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:09.689785 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" event={"ID":"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46","Type":"ContainerStarted","Data":"e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a"} Oct 30 06:19:09.690240 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:09.689838 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" event={"ID":"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46","Type":"ContainerStarted","Data":"c919801cf527a5a97410e5ea5fdda3456b5c3d62f2e67d48d84e74bfe3c8f9c1"} Oct 30 06:19:09.690240 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:09.689913 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:09.732041 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:09.731983 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" podStartSLOduration=1.731968584 podStartE2EDuration="1.731968584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:09.728051867 +0000 UTC m=+2909.331741125" watchObservedRunningTime="2026-10-30 06:19:09.731968584 +0000 UTC m=+2909.335657883" Oct 30 06:19:20.373900 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.373816 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc"] Oct 30 06:19:20.379698 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.379678 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.393800 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.393772 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a8b892-serving-cert\"" Oct 30 06:19:20.411439 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.411393 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc"] Oct 30 06:19:20.430602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.429912 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7w6\" (UniqueName: \"kubernetes.io/projected/76a5be12-b842-446d-b841-3af5bf5dde44-kube-api-access-zj7w6\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.430602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.430090 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a5be12-b842-446d-b841-3af5bf5dde44-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.430602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.430139 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.430602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.430274 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.535853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.535809 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.536057 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.535970 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7w6\" (UniqueName: \"kubernetes.io/projected/76a5be12-b842-446d-b841-3af5bf5dde44-kube-api-access-zj7w6\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.536179 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.536159 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a5be12-b842-446d-b841-3af5bf5dde44-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.536246 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.536212 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.538155 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.538121 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a5be12-b842-446d-b841-3af5bf5dde44-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.566446 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.566400 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.611439 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.611378 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7w6\" (UniqueName: \"kubernetes.io/projected/76a5be12-b842-446d-b841-3af5bf5dde44-kube-api-access-zj7w6\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.617538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.612119 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.698420 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.698336 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:20.700009 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:20.699987 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:19:21.025201 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:21.025170 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc"] Oct 30 06:19:21.026690 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:19:21.026659 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a5be12_b842_446d_b841_3af5bf5dde44.slice/crio-2586d433e18eea7e15a55fdda3d0090273fd243cda25abe1aa662a1e9415690b WatchSource:0}: Error finding container 2586d433e18eea7e15a55fdda3d0090273fd243cda25abe1aa662a1e9415690b: Status 404 returned error can't find the container with id 2586d433e18eea7e15a55fdda3d0090273fd243cda25abe1aa662a1e9415690b Oct 30 06:19:21.791306 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:21.791213 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" event={"ID":"76a5be12-b842-446d-b841-3af5bf5dde44","Type":"ContainerStarted","Data":"5e8e7e3e2aa631876bc484aa80f96416982f65f5f97d78d56680fb9eb29a0824"} Oct 30 06:19:21.791306 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:21.791276 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" event={"ID":"76a5be12-b842-446d-b841-3af5bf5dde44","Type":"ContainerStarted","Data":"2586d433e18eea7e15a55fdda3d0090273fd243cda25abe1aa662a1e9415690b"} Oct 30 06:19:21.791781 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:21.791422 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:21.835032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:21.834976 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" podStartSLOduration=1.447457749 podStartE2EDuration="1.834960358s" totalImagesPullingTime="387.502609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:21.83173414 +0000 UTC m=+2921.435423398" watchObservedRunningTime="2026-10-30 06:19:21.834960358 +0000 UTC m=+2921.438649612" Oct 30 06:19:27.836354 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:27.835414 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:19:30.717493 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:19:30.717436 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:19:52.850383 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:52.850340 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2"] Oct 30 06:19:52.851061 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:52.850589 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" podUID="d02b2f01-c051-4989-bbc8-f9d174762e2f" containerName="maas-api" containerID="cri-o://e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332" gracePeriod=30 Oct 30 06:19:53.173999 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.173968 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:19:53.294629 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.294596 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02b2f01-c051-4989-bbc8-f9d174762e2f-kube-api-access-n8nf7\" (UniqueName: \"kubernetes.io/projected/d02b2f01-c051-4989-bbc8-f9d174762e2f-kube-api-access-n8nf7\") pod \"d02b2f01-c051-4989-bbc8-f9d174762e2f\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " Oct 30 06:19:53.294835 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.294711 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-metrics-certs\") pod \"d02b2f01-c051-4989-bbc8-f9d174762e2f\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " Oct 30 06:19:53.294926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.294862 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02b2f01-c051-4989-bbc8-f9d174762e2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02b2f01-c051-4989-bbc8-f9d174762e2f-openshift-service-ca\") pod \"d02b2f01-c051-4989-bbc8-f9d174762e2f\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " Oct 30 06:19:53.294992 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.294935 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-maas-api-tls\") pod \"d02b2f01-c051-4989-bbc8-f9d174762e2f\" (UID: \"d02b2f01-c051-4989-bbc8-f9d174762e2f\") " Oct 30 06:19:53.295245 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.295215 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02b2f01-c051-4989-bbc8-f9d174762e2f-openshift-service-ca" pod "d02b2f01-c051-4989-bbc8-f9d174762e2f" (UID: "d02b2f01-c051-4989-bbc8-f9d174762e2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:53.295606 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.295587 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02b2f01-c051-4989-bbc8-f9d174762e2f-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:19:53.296169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.296127 5072 generic.go:356] "Generic (PLEG): container finished" podID="d02b2f01-c051-4989-bbc8-f9d174762e2f" containerID="e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332" exitCode=0 Oct 30 06:19:53.296293 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.296206 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" Oct 30 06:19:53.296293 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.296268 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" event={"ID":"d02b2f01-c051-4989-bbc8-f9d174762e2f","Type":"ContainerDied","Data":"e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332"} Oct 30 06:19:53.296413 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.296319 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2" event={"ID":"d02b2f01-c051-4989-bbc8-f9d174762e2f","Type":"ContainerDied","Data":"21fd878eca69d50b7e310b58e4a42007f7c186d9d756527fa84ac4b81d46b619"} Oct 30 06:19:53.296413 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.296348 5072 scope.go:122] "RemoveContainer" containerID="e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332" Oct 30 06:19:53.297230 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.297204 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-metrics-certs" pod "d02b2f01-c051-4989-bbc8-f9d174762e2f" (UID: "d02b2f01-c051-4989-bbc8-f9d174762e2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:53.297334 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.297306 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02b2f01-c051-4989-bbc8-f9d174762e2f-kube-api-access-n8nf7" pod "d02b2f01-c051-4989-bbc8-f9d174762e2f" (UID: "d02b2f01-c051-4989-bbc8-f9d174762e2f"). InnerVolumeSpecName "kube-api-access-n8nf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:53.297405 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.297358 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-maas-api-tls" pod "d02b2f01-c051-4989-bbc8-f9d174762e2f" (UID: "d02b2f01-c051-4989-bbc8-f9d174762e2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:53.411533 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.398156 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:19:53.411533 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.398196 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8nf7\" (UniqueName: \"kubernetes.io/projected/d02b2f01-c051-4989-bbc8-f9d174762e2f-kube-api-access-n8nf7\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:19:53.411533 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.398214 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02b2f01-c051-4989-bbc8-f9d174762e2f-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:19:53.474433 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.469951 5072 scope.go:122] "RemoveContainer" containerID="e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332" Oct 30 06:19:53.513499 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:19:53.504059 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332\": container with ID starting with e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332 not found: ID does not exist" containerID="e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332" Oct 30 06:19:53.513499 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.504104 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332"} err="failed to get container status \"e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332\": rpc error: code = NotFound desc = could not find container \"e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332\": container with ID starting with e893924bca1fa4195fc1644bc5951e74f864086194ab07a35d45bfa436ecb332 not found: ID does not exist" Oct 30 06:19:53.667550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.666827 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2"] Oct 30 06:19:53.672316 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:53.672289 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de2bf2-586bfd985c-xdzp2"] Oct 30 06:19:54.522430 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:54.522394 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02b2f01-c051-4989-bbc8-f9d174762e2f" path="/var/lib/kubelet/pods/d02b2f01-c051-4989-bbc8-f9d174762e2f/volumes" Oct 30 06:19:59.842698 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:59.842664 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm"] Oct 30 06:19:59.843694 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:59.843668 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d02b2f01-c051-4989-bbc8-f9d174762e2f" containerName="maas-api" Oct 30 06:19:59.843694 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:59.843693 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02b2f01-c051-4989-bbc8-f9d174762e2f" containerName="maas-api" Oct 30 06:19:59.843966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:59.843950 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02b2f01-c051-4989-bbc8-f9d174762e2f" containerName="maas-api" Oct 30 06:19:59.850075 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:59.849757 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:19:59.869037 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:59.869013 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-718730-openshift-default-dockercfg-z2t6s\"" Oct 30 06:19:59.888555 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:19:59.888524 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm"] Oct 30 06:20:00.030485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.028531 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-token\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.030485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.028697 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-envoy\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.030485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.028803 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-podinfo\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.030485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.028851 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-socket\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.030485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.028931 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-credential-socket\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.030485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.028958 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-data\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.030485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.028986 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl4kl\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-kube-api-access-rl4kl\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.030485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.029015 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-certs\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.030485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.029082 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8ac2d-724c-4309-aaba-de0952fc9802-istiod-ca-cert\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.142484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.141836 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-envoy\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.142484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.141970 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-podinfo\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.142484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.142020 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-socket\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.142484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.142098 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-credential-socket\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.142484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.142123 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-data\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.142484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.142153 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4kl\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-kube-api-access-rl4kl\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.142484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.142182 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-certs\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.142484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.142244 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8ac2d-724c-4309-aaba-de0952fc9802-istiod-ca-cert\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.142484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.142346 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-token\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.151545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.143621 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-data\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.151545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.144280 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-certs\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.177477 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.173819 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8ac2d-724c-4309-aaba-de0952fc9802-istiod-ca-cert\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.177477 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.175077 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-socket\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.177477 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.175407 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-credential-socket\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.190542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.188795 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-envoy\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.191516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.191479 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-token\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.206372 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.206340 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4kl\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-kube-api-access-rl4kl\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.227167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.227100 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-podinfo\") pod \"e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.469362 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.469272 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:00.721117 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:20:00.719355 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:20:00.800874 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.800844 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm"] Oct 30 06:20:00.800874 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:20:00.800857 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf8ac2d_724c_4309_aaba_de0952fc9802.slice/crio-8a6d0b220399f39a1520db875140dded33be5c15d28fc5ac73f65146dd1bfbbe WatchSource:0}: Error finding container 8a6d0b220399f39a1520db875140dded33be5c15d28fc5ac73f65146dd1bfbbe: Status 404 returned error can't find the container with id 8a6d0b220399f39a1520db875140dded33be5c15d28fc5ac73f65146dd1bfbbe Oct 30 06:20:00.803833 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.803801 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:20:00.803938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.803866 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:20:00.803938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:00.803895 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:20:01.139475 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.139424 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf"] Oct 30 06:20:01.139943 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.139663 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" containerID="cri-o://127f4f650bebfbe066e7f2b34a5ffdcce6544397adf5a5ba3de38df29fa80119" gracePeriod=30 Oct 30 06:20:01.147690 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.147666 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt"] Oct 30 06:20:01.147873 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.147849 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" containerID="cri-o://d1dfe6f5c9e86cab69221ca26d0bbf2d96d6d30a33214b229cd3c0173e460229" gracePeriod=30 Oct 30 06:20:01.410587 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.403308 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" event={"ID":"3bf8ac2d-724c-4309-aaba-de0952fc9802","Type":"ContainerStarted","Data":"6ca4efaf2d9c50e7b0393d48c3803f210bc380f94e8375273aea3246fc41d33a"} Oct 30 06:20:01.410587 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.403384 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" event={"ID":"3bf8ac2d-724c-4309-aaba-de0952fc9802","Type":"ContainerStarted","Data":"8a6d0b220399f39a1520db875140dded33be5c15d28fc5ac73f65146dd1bfbbe"} Oct 30 06:20:01.460697 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.460639 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podStartSLOduration=2.460622444 podStartE2EDuration="2.460622444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:01.457789837 +0000 UTC m=+2961.061479107" watchObservedRunningTime="2026-10-30 06:20:01.460622444 +0000 UTC m=+2961.064311701" Oct 30 06:20:01.470588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.470547 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:01.471862 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.471836 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:20:01.471937 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:01.471893 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:20:02.612481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:02.605808 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:03.485768 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:03.485735 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:03.486740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:03.486722 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:20:06.573502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.569849 5072 generic.go:356] "Generic (PLEG): container finished" podID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerID="127f4f650bebfbe066e7f2b34a5ffdcce6544397adf5a5ba3de38df29fa80119" exitCode=0 Oct 30 06:20:06.573502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.569926 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" event={"ID":"567a0a2d-a294-4b2d-9a0b-ddc235184e72","Type":"ContainerDied","Data":"127f4f650bebfbe066e7f2b34a5ffdcce6544397adf5a5ba3de38df29fa80119"} Oct 30 06:20:06.573502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.569993 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" event={"ID":"567a0a2d-a294-4b2d-9a0b-ddc235184e72","Type":"ContainerDied","Data":"ca9d1bcda9e020073d274c0dbf6c69d8fb37faef12b545ec6e57a01ed70d106f"} Oct 30 06:20:06.573502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.570019 5072 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca9d1bcda9e020073d274c0dbf6c69d8fb37faef12b545ec6e57a01ed70d106f" Oct 30 06:20:06.573502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.572701 5072 generic.go:356] "Generic (PLEG): container finished" podID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerID="d1dfe6f5c9e86cab69221ca26d0bbf2d96d6d30a33214b229cd3c0173e460229" exitCode=0 Oct 30 06:20:06.573502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.572783 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" event={"ID":"d5fe0fd5-460a-4faf-854c-8172f9e00e2b","Type":"ContainerDied","Data":"d1dfe6f5c9e86cab69221ca26d0bbf2d96d6d30a33214b229cd3c0173e460229"} Oct 30 06:20:06.573502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.572820 5072 scope.go:122] "RemoveContainer" containerID="53ece7de56395ebdce7ba495df676642dee33ee9cf0e7d865de90da2cc5a0447" Oct 30 06:20:06.576641 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.576621 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:20:06.703843 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.703810 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:20:06.768525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.768376 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-socket\") pod \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " Oct 30 06:20:06.768788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.768764 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-envoy\") pod \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " Oct 30 06:20:06.769009 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.768993 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-certs\") pod \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " Oct 30 06:20:06.769206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.769187 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-podinfo\") pod \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " Oct 30 06:20:06.769346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.769330 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-credential-socket\") pod \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " Oct 30 06:20:06.769590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.769569 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-kube-api-access-9k2g5\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-kube-api-access-9k2g5\") pod \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " Oct 30 06:20:06.769839 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.769811 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-token\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-token\") pod \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " Oct 30 06:20:06.770010 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.769992 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-data\") pod \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " Oct 30 06:20:06.770197 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.770178 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-token\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-token\") pod \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " Oct 30 06:20:06.770333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.770318 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istiod-ca-cert\") pod \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " Oct 30 06:20:06.770524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.770505 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-podinfo\") pod \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " Oct 30 06:20:06.770738 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.770700 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istiod-ca-cert\") pod \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\" (UID: \"567a0a2d-a294-4b2d-9a0b-ddc235184e72\") " Oct 30 06:20:06.772170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.772146 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istiod-ca-cert" pod "567a0a2d-a294-4b2d-9a0b-ddc235184e72" (UID: "567a0a2d-a294-4b2d-9a0b-ddc235184e72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:06.775565 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.775358 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-podinfo" pod "567a0a2d-a294-4b2d-9a0b-ddc235184e72" (UID: "567a0a2d-a294-4b2d-9a0b-ddc235184e72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:06.775999 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.775717 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-credential-socket" pod "567a0a2d-a294-4b2d-9a0b-ddc235184e72" (UID: "567a0a2d-a294-4b2d-9a0b-ddc235184e72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.777582 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.777557 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-certs" pod "567a0a2d-a294-4b2d-9a0b-ddc235184e72" (UID: "567a0a2d-a294-4b2d-9a0b-ddc235184e72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.777767 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.777729 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-socket" pod "567a0a2d-a294-4b2d-9a0b-ddc235184e72" (UID: "567a0a2d-a294-4b2d-9a0b-ddc235184e72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.779333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.779101 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-kube-api-access-9k2g5" pod "567a0a2d-a294-4b2d-9a0b-ddc235184e72" (UID: "567a0a2d-a294-4b2d-9a0b-ddc235184e72"). InnerVolumeSpecName "kube-api-access-9k2g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:06.780658 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.779615 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istiod-ca-cert" pod "d5fe0fd5-460a-4faf-854c-8172f9e00e2b" (UID: "d5fe0fd5-460a-4faf-854c-8172f9e00e2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:06.780658 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.780632 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-data" pod "567a0a2d-a294-4b2d-9a0b-ddc235184e72" (UID: "567a0a2d-a294-4b2d-9a0b-ddc235184e72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.782060 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.782037 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-token" pod "d5fe0fd5-460a-4faf-854c-8172f9e00e2b" (UID: "d5fe0fd5-460a-4faf-854c-8172f9e00e2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:06.782163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.782116 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-token" pod "567a0a2d-a294-4b2d-9a0b-ddc235184e72" (UID: "567a0a2d-a294-4b2d-9a0b-ddc235184e72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:06.783072 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.783045 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-envoy" pod "567a0a2d-a294-4b2d-9a0b-ddc235184e72" (UID: "567a0a2d-a294-4b2d-9a0b-ddc235184e72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.784074 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.784037 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-podinfo" pod "d5fe0fd5-460a-4faf-854c-8172f9e00e2b" (UID: "d5fe0fd5-460a-4faf-854c-8172f9e00e2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:06.873098 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873063 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-kube-api-access-xl9q8\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-kube-api-access-xl9q8\") pod \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " Oct 30 06:20:06.873265 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873157 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-envoy\") pod \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " Oct 30 06:20:06.873265 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873251 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-socket\") pod \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " Oct 30 06:20:06.873343 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873325 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-credential-socket\") pod \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " Oct 30 06:20:06.873384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873362 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-data\") pod \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " Oct 30 06:20:06.873431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873415 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-certs\") pod \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\" (UID: \"d5fe0fd5-460a-4faf-854c-8172f9e00e2b\") " Oct 30 06:20:06.873602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873573 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-socket" pod "d5fe0fd5-460a-4faf-854c-8172f9e00e2b" (UID: "d5fe0fd5-460a-4faf-854c-8172f9e00e2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.873833 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873749 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-credential-socket" pod "d5fe0fd5-460a-4faf-854c-8172f9e00e2b" (UID: "d5fe0fd5-460a-4faf-854c-8172f9e00e2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.873833 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873823 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-certs" pod "d5fe0fd5-460a-4faf-854c-8172f9e00e2b" (UID: "d5fe0fd5-460a-4faf-854c-8172f9e00e2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.874084 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.873946 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-data" pod "d5fe0fd5-460a-4faf-854c-8172f9e00e2b" (UID: "d5fe0fd5-460a-4faf-854c-8172f9e00e2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.874369 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874349 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874370 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874379 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874387 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874395 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874405 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874414 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874423 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k2g5\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-kube-api-access-9k2g5\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874431 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874439 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874446 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.874448 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874453 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.875081 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874481 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.875081 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874493 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.875081 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874501 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.875081 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.874508 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567a0a2d-a294-4b2d-9a0b-ddc235184e72-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.875341 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.875320 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-envoy" pod "d5fe0fd5-460a-4faf-854c-8172f9e00e2b" (UID: "d5fe0fd5-460a-4faf-854c-8172f9e00e2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.875655 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.875638 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-kube-api-access-xl9q8" pod "d5fe0fd5-460a-4faf-854c-8172f9e00e2b" (UID: "d5fe0fd5-460a-4faf-854c-8172f9e00e2b"). InnerVolumeSpecName "kube-api-access-xl9q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:06.978049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.977935 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl9q8\" (UniqueName: \"kubernetes.io/projected/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-kube-api-access-xl9q8\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.978049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:06.977976 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5fe0fd5-460a-4faf-854c-8172f9e00e2b-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:07.581669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:07.581579 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" event={"ID":"d5fe0fd5-460a-4faf-854c-8172f9e00e2b","Type":"ContainerDied","Data":"a8538169cc627e5a266b046f8a94caf5188ca4324197ab117320113b07c2b9c9"} Oct 30 06:20:07.581669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:07.581605 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf" Oct 30 06:20:07.581669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:07.581650 5072 scope.go:122] "RemoveContainer" containerID="d1dfe6f5c9e86cab69221ca26d0bbf2d96d6d30a33214b229cd3c0173e460229" Oct 30 06:20:07.582286 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:07.581605 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt" Oct 30 06:20:07.660630 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:07.660592 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf"] Oct 30 06:20:07.719569 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:07.719508 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-tw7bf"] Oct 30 06:20:07.737514 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:07.737425 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt"] Oct 30 06:20:07.760558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:07.759155 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-8qzzt"] Oct 30 06:20:08.553354 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:08.553321 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" path="/var/lib/kubelet/pods/567a0a2d-a294-4b2d-9a0b-ddc235184e72/volumes" Oct 30 06:20:08.554288 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:08.554269 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" path="/var/lib/kubelet/pods/d5fe0fd5-460a-4faf-854c-8172f9e00e2b/volumes" Oct 30 06:20:12.080545 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:12.080508 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft"] Oct 30 06:20:14.688317 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:14.688281 5072 generic.go:356] "Generic (PLEG): container finished" podID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerID="97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e" exitCode=0 Oct 30 06:20:14.688806 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:14.688356 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" event={"ID":"29d97d10-ff5d-499a-a994-2721e2e3c8fa","Type":"ContainerDied","Data":"97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e"} Oct 30 06:20:14.688859 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:14.688847 5072 scope.go:122] "RemoveContainer" containerID="97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e" Oct 30 06:20:14.690952 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:14.690920 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:20:14.691069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:14.690994 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:20:14.691069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:14.691029 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:20:15.697912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:15.697878 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" event={"ID":"29d97d10-ff5d-499a-a994-2721e2e3c8fa","Type":"ContainerStarted","Data":"c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f"} Oct 30 06:20:16.336994 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:16.336961 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:20:16.337377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:16.337351 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 06:20:16.337585 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:16.337560 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 06:20:17.081031 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:17.080986 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" podUID="51a6043d-0206-452f-92a8-3a6324d213aa" containerName="payload-processing" containerID="cri-o://83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e" gracePeriod=25 Oct 30 06:20:17.341072 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:17.340557 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 06:20:17.341072 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:17.340622 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 06:20:18.334495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:18.334413 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 06:20:18.335003 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:18.334513 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 06:20:19.336008 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:19.335965 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 06:20:19.336631 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:19.336046 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 06:20:20.335581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:20.335334 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 06:20:20.335581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:20.335402 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 06:20:21.339556 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:21.339516 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:20:21.339969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:21.339909 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:20:21.340627 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:21.340605 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:20:22.657826 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:22.657792 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f"] Oct 30 06:20:22.658346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:22.658022 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" containerID="cri-o://2b8a945c1d056cfdeb5b80c265c316ffed2c526bdc9a72f23e6ef5e0526440d4" gracePeriod=30 Oct 30 06:20:23.257181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:23.257135 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" podUID="51a6043d-0206-452f-92a8-3a6324d213aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:20:27.822878 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:27.822839 5072 generic.go:356] "Generic (PLEG): container finished" podID="11307d4e-323b-4246-9903-c9eab92335a7" containerID="2b8a945c1d056cfdeb5b80c265c316ffed2c526bdc9a72f23e6ef5e0526440d4" exitCode=0 Oct 30 06:20:27.823430 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:27.822895 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" event={"ID":"11307d4e-323b-4246-9903-c9eab92335a7","Type":"ContainerDied","Data":"2b8a945c1d056cfdeb5b80c265c316ffed2c526bdc9a72f23e6ef5e0526440d4"} Oct 30 06:20:27.958859 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:27.958831 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:20:28.112815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.112775 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11307d4e-323b-4246-9903-c9eab92335a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11307d4e-323b-4246-9903-c9eab92335a7-istiod-ca-cert\") pod \"11307d4e-323b-4246-9903-c9eab92335a7\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " Oct 30 06:20:28.113049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.112854 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-istio-token\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-istio-token\") pod \"11307d4e-323b-4246-9903-c9eab92335a7\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " Oct 30 06:20:28.113049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.112938 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-socket\") pod \"11307d4e-323b-4246-9903-c9eab92335a7\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " Oct 30 06:20:28.113049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113031 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-data\") pod \"11307d4e-323b-4246-9903-c9eab92335a7\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " Oct 30 06:20:28.113247 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113095 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-certs\") pod \"11307d4e-323b-4246-9903-c9eab92335a7\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " Oct 30 06:20:28.113247 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113124 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-credential-socket\") pod \"11307d4e-323b-4246-9903-c9eab92335a7\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " Oct 30 06:20:28.113247 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113152 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11307d4e-323b-4246-9903-c9eab92335a7-istiod-ca-cert" pod "11307d4e-323b-4246-9903-c9eab92335a7" (UID: "11307d4e-323b-4246-9903-c9eab92335a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:28.113247 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113255 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-envoy\") pod \"11307d4e-323b-4246-9903-c9eab92335a7\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113290 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11307d4e-323b-4246-9903-c9eab92335a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11307d4e-323b-4246-9903-c9eab92335a7-istio-podinfo\") pod \"11307d4e-323b-4246-9903-c9eab92335a7\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113348 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-socket" pod "11307d4e-323b-4246-9903-c9eab92335a7" (UID: "11307d4e-323b-4246-9903-c9eab92335a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113390 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-kube-api-access-4jvnx\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-kube-api-access-4jvnx\") pod \"11307d4e-323b-4246-9903-c9eab92335a7\" (UID: \"11307d4e-323b-4246-9903-c9eab92335a7\") " Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113549 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-certs" pod "11307d4e-323b-4246-9903-c9eab92335a7" (UID: "11307d4e-323b-4246-9903-c9eab92335a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113577 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-data" pod "11307d4e-323b-4246-9903-c9eab92335a7" (UID: "11307d4e-323b-4246-9903-c9eab92335a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.113747 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-credential-socket" pod "11307d4e-323b-4246-9903-c9eab92335a7" (UID: "11307d4e-323b-4246-9903-c9eab92335a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.114137 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.114161 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.114174 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.114187 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11307d4e-323b-4246-9903-c9eab92335a7-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.114198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.114201 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.116087 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.116054 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-istio-token" pod "11307d4e-323b-4246-9903-c9eab92335a7" (UID: "11307d4e-323b-4246-9903-c9eab92335a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:28.116306 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.116282 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11307d4e-323b-4246-9903-c9eab92335a7-istio-podinfo" pod "11307d4e-323b-4246-9903-c9eab92335a7" (UID: "11307d4e-323b-4246-9903-c9eab92335a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:28.118596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.118562 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-envoy" pod "11307d4e-323b-4246-9903-c9eab92335a7" (UID: "11307d4e-323b-4246-9903-c9eab92335a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.118697 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.118664 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-kube-api-access-4jvnx" pod "11307d4e-323b-4246-9903-c9eab92335a7" (UID: "11307d4e-323b-4246-9903-c9eab92335a7"). InnerVolumeSpecName "kube-api-access-4jvnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:28.215778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.215688 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.215778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.215729 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11307d4e-323b-4246-9903-c9eab92335a7-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.215778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.215745 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11307d4e-323b-4246-9903-c9eab92335a7-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.215778 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.215760 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jvnx\" (UniqueName: \"kubernetes.io/projected/11307d4e-323b-4246-9903-c9eab92335a7-kube-api-access-4jvnx\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.831321 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.831242 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" Oct 30 06:20:28.831850 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.831234 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f" event={"ID":"11307d4e-323b-4246-9903-c9eab92335a7","Type":"ContainerDied","Data":"cf543e4af74c6b30e6382275a46f6576e8e1d5242e8064bae4da1cfa790f0dfe"} Oct 30 06:20:28.831850 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.831387 5072 scope.go:122] "RemoveContainer" containerID="2b8a945c1d056cfdeb5b80c265c316ffed2c526bdc9a72f23e6ef5e0526440d4" Oct 30 06:20:28.892848 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.892813 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f"] Oct 30 06:20:28.903532 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:28.903497 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02ceed-openshift-default-6c498496dc-kzp8f"] Oct 30 06:20:30.521167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:30.521132 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11307d4e-323b-4246-9903-c9eab92335a7" path="/var/lib/kubelet/pods/11307d4e-323b-4246-9903-c9eab92335a7/volumes" Oct 30 06:20:30.711447 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:20:30.711405 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:20:33.253333 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:33.253296 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" podUID="51a6043d-0206-452f-92a8-3a6324d213aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:20:34.826337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:34.826297 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp"] Oct 30 06:20:34.826872 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:34.826691 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" podUID="68292eb0-96f8-4e6f-9668-3dd4b531496a" containerName="maas-api" containerID="cri-o://22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86" gracePeriod=30 Oct 30 06:20:35.417343 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.417010 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 06:20:35.536053 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.536024 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68292eb0-96f8-4e6f-9668-3dd4b531496a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68292eb0-96f8-4e6f-9668-3dd4b531496a-openshift-service-ca\") pod \"68292eb0-96f8-4e6f-9668-3dd4b531496a\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " Oct 30 06:20:35.536259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.536077 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-metrics-certs\") pod \"68292eb0-96f8-4e6f-9668-3dd4b531496a\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " Oct 30 06:20:35.536259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.536250 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68292eb0-96f8-4e6f-9668-3dd4b531496a-kube-api-access-h9lp8\" (UniqueName: \"kubernetes.io/projected/68292eb0-96f8-4e6f-9668-3dd4b531496a-kube-api-access-h9lp8\") pod \"68292eb0-96f8-4e6f-9668-3dd4b531496a\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " Oct 30 06:20:35.536377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.536292 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-maas-api-tls\") pod \"68292eb0-96f8-4e6f-9668-3dd4b531496a\" (UID: \"68292eb0-96f8-4e6f-9668-3dd4b531496a\") " Oct 30 06:20:35.536436 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.536389 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68292eb0-96f8-4e6f-9668-3dd4b531496a-openshift-service-ca" pod "68292eb0-96f8-4e6f-9668-3dd4b531496a" (UID: "68292eb0-96f8-4e6f-9668-3dd4b531496a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:35.537321 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.537290 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68292eb0-96f8-4e6f-9668-3dd4b531496a-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.538626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.538597 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-maas-api-tls" pod "68292eb0-96f8-4e6f-9668-3dd4b531496a" (UID: "68292eb0-96f8-4e6f-9668-3dd4b531496a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:35.538731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.538671 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-metrics-certs" pod "68292eb0-96f8-4e6f-9668-3dd4b531496a" (UID: "68292eb0-96f8-4e6f-9668-3dd4b531496a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:35.538776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.538751 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68292eb0-96f8-4e6f-9668-3dd4b531496a-kube-api-access-h9lp8" pod "68292eb0-96f8-4e6f-9668-3dd4b531496a" (UID: "68292eb0-96f8-4e6f-9668-3dd4b531496a"). InnerVolumeSpecName "kube-api-access-h9lp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:35.638998 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.638963 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.638998 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.639001 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9lp8\" (UniqueName: \"kubernetes.io/projected/68292eb0-96f8-4e6f-9668-3dd4b531496a-kube-api-access-h9lp8\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.639278 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.639017 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68292eb0-96f8-4e6f-9668-3dd4b531496a-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.907363 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.907274 5072 generic.go:356] "Generic (PLEG): container finished" podID="68292eb0-96f8-4e6f-9668-3dd4b531496a" containerID="22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86" exitCode=0 Oct 30 06:20:35.907975 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.907949 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" event={"ID":"68292eb0-96f8-4e6f-9668-3dd4b531496a","Type":"ContainerDied","Data":"22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86"} Oct 30 06:20:35.908113 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.908099 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" event={"ID":"68292eb0-96f8-4e6f-9668-3dd4b531496a","Type":"ContainerDied","Data":"544fe7fdd05a9e438098a8a41cd525794c59d5e3a1a9b481ab1b7b98d32e1167"} Oct 30 06:20:35.908237 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.908224 5072 scope.go:122] "RemoveContainer" containerID="22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86" Oct 30 06:20:35.908504 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.908487 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp" Oct 30 06:20:35.932099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.932071 5072 scope.go:122] "RemoveContainer" containerID="22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86" Oct 30 06:20:35.933417 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:20:35.933392 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86\": container with ID starting with 22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86 not found: ID does not exist" containerID="22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86" Oct 30 06:20:35.933833 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.933426 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86"} err="failed to get container status \"22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86\": rpc error: code = NotFound desc = could not find container \"22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86\": container with ID starting with 22be7732e911819c5a64ec6855bf6ba541d0ade186d7099c867269a365240c86 not found: ID does not exist" Oct 30 06:20:35.966967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.966936 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp"] Oct 30 06:20:35.972526 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:35.972499 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7a506-6b5fc59c6b-zpgkp"] Oct 30 06:20:36.520249 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:36.520218 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68292eb0-96f8-4e6f-9668-3dd4b531496a" path="/var/lib/kubelet/pods/68292eb0-96f8-4e6f-9668-3dd4b531496a/volumes" Oct 30 06:20:37.405284 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.405256 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 06:20:37.470178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.470137 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a6043d-0206-452f-92a8-3a6324d213aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a6043d-0206-452f-92a8-3a6324d213aa-plugins-config-volume\") pod \"51a6043d-0206-452f-92a8-3a6324d213aa\" (UID: \"51a6043d-0206-452f-92a8-3a6324d213aa\") " Oct 30 06:20:37.470428 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.470399 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a6043d-0206-452f-92a8-3a6324d213aa-kube-api-access-sh6pd\" (UniqueName: \"kubernetes.io/projected/51a6043d-0206-452f-92a8-3a6324d213aa-kube-api-access-sh6pd\") pod \"51a6043d-0206-452f-92a8-3a6324d213aa\" (UID: \"51a6043d-0206-452f-92a8-3a6324d213aa\") " Oct 30 06:20:37.471169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.471122 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a6043d-0206-452f-92a8-3a6324d213aa-plugins-config-volume" pod "51a6043d-0206-452f-92a8-3a6324d213aa" (UID: "51a6043d-0206-452f-92a8-3a6324d213aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:37.474340 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.474313 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a6043d-0206-452f-92a8-3a6324d213aa-kube-api-access-sh6pd" pod "51a6043d-0206-452f-92a8-3a6324d213aa" (UID: "51a6043d-0206-452f-92a8-3a6324d213aa"). InnerVolumeSpecName "kube-api-access-sh6pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:37.572365 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.572327 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6pd\" (UniqueName: \"kubernetes.io/projected/51a6043d-0206-452f-92a8-3a6324d213aa-kube-api-access-sh6pd\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:37.572365 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.572353 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a6043d-0206-452f-92a8-3a6324d213aa-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:37.926770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.926734 5072 generic.go:356] "Generic (PLEG): container finished" podID="51a6043d-0206-452f-92a8-3a6324d213aa" containerID="83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e" exitCode=0 Oct 30 06:20:37.926995 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.926781 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" Oct 30 06:20:37.926995 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.926816 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" event={"ID":"51a6043d-0206-452f-92a8-3a6324d213aa","Type":"ContainerDied","Data":"83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e"} Oct 30 06:20:37.926995 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.926869 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft" event={"ID":"51a6043d-0206-452f-92a8-3a6324d213aa","Type":"ContainerDied","Data":"36c74ffad0d9cb7e8d28f747b6d9fec94196f95829e23cd26a56be11a29948e9"} Oct 30 06:20:37.926995 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.926891 5072 scope.go:122] "RemoveContainer" containerID="83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e" Oct 30 06:20:37.940176 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.940141 5072 scope.go:122] "RemoveContainer" containerID="83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e" Oct 30 06:20:37.940718 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:20:37.940687 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e\": container with ID starting with 83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e not found: ID does not exist" containerID="83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e" Oct 30 06:20:37.940814 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.940723 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e"} err="failed to get container status \"83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e\": rpc error: code = NotFound desc = could not find container \"83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e\": container with ID starting with 83d162af9ac9c298c054745378b8c24a61f77bbe772bb7bc883a0763a301120e not found: ID does not exist" Oct 30 06:20:37.977878 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.977839 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft"] Oct 30 06:20:37.990530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:37.990482 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-1b4b7a-5797f6c7fd-hcfft"] Oct 30 06:20:38.631021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:38.630981 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a6043d-0206-452f-92a8-3a6324d213aa" path="/var/lib/kubelet/pods/51a6043d-0206-452f-92a8-3a6324d213aa/volumes" Oct 30 06:20:42.895483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:42.876962 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2"] Oct 30 06:20:42.895483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:42.877243 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" containerID="cri-o://43437b170d9900a2d5f67edd8a8ea89bdaed43cddf1a7a73f4a2e2820970c3bb" gracePeriod=30 Oct 30 06:20:44.438596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.438557 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl"] Oct 30 06:20:44.439409 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439389 5072 cpu_manager.go:414] "containerMap: removing container" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" Oct 30 06:20:44.439530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439410 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" Oct 30 06:20:44.439530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439430 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" Oct 30 06:20:44.439530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439436 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" Oct 30 06:20:44.439530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439447 5072 cpu_manager.go:414] "containerMap: removing container" podUID="51a6043d-0206-452f-92a8-3a6324d213aa" containerName="payload-processing" Oct 30 06:20:44.439530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439452 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a6043d-0206-452f-92a8-3a6324d213aa" containerName="payload-processing" Oct 30 06:20:44.439530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439503 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" Oct 30 06:20:44.439530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439509 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" Oct 30 06:20:44.439530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439526 5072 cpu_manager.go:414] "containerMap: removing container" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" Oct 30 06:20:44.439530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439530 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" Oct 30 06:20:44.440033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439544 5072 cpu_manager.go:414] "containerMap: removing container" podUID="68292eb0-96f8-4e6f-9668-3dd4b531496a" containerName="maas-api" Oct 30 06:20:44.440033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439548 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68292eb0-96f8-4e6f-9668-3dd4b531496a" containerName="maas-api" Oct 30 06:20:44.440033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439707 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="68292eb0-96f8-4e6f-9668-3dd4b531496a" containerName="maas-api" Oct 30 06:20:44.440033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439731 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="567a0a2d-a294-4b2d-9a0b-ddc235184e72" containerName="istio-proxy" Oct 30 06:20:44.440033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439742 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" Oct 30 06:20:44.440033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439750 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a6043d-0206-452f-92a8-3a6324d213aa" containerName="payload-processing" Oct 30 06:20:44.440033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439761 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fe0fd5-460a-4faf-854c-8172f9e00e2b" containerName="istio-proxy" Oct 30 06:20:44.440033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.439768 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="11307d4e-323b-4246-9903-c9eab92335a7" containerName="istio-proxy" Oct 30 06:20:44.443875 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.443856 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:20:44.473058 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.473030 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a8b892-dockercfg-pfnrr\"" Oct 30 06:20:44.473258 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.473108 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a8b892\"" Oct 30 06:20:44.477528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.477502 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl"] Oct 30 06:20:44.486867 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.486831 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl\" (UID: \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:20:44.487202 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.487181 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrqc\" (UniqueName: \"kubernetes.io/projected/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-kube-api-access-hqrqc\") pod \"payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl\" (UID: \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:20:44.588311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.588276 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl\" (UID: \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:20:44.588588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.588564 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrqc\" (UniqueName: \"kubernetes.io/projected/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-kube-api-access-hqrqc\") pod \"payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl\" (UID: \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:20:44.589262 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.589237 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl\" (UID: \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:20:44.610886 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.610857 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrqc\" (UniqueName: \"kubernetes.io/projected/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-kube-api-access-hqrqc\") pod \"payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl\" (UID: \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:20:44.760132 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.760030 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:20:44.822974 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.822941 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc"] Oct 30 06:20:44.848793 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.848762 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:44.865678 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.865647 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-718730-serving-cert\"" Oct 30 06:20:44.882406 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.882380 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc"] Oct 30 06:20:44.894330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.893022 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:44.894330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.893123 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af021843-9c8f-4ce6-8f08-858104de58dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:44.894330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.893163 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:44.894330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:44.893528 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gb98\" (UniqueName: \"kubernetes.io/projected/af021843-9c8f-4ce6-8f08-858104de58dc-kube-api-access-6gb98\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.054586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.047718 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.054586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.047861 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af021843-9c8f-4ce6-8f08-858104de58dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.054586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.047912 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.054586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.048503 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gb98\" (UniqueName: \"kubernetes.io/projected/af021843-9c8f-4ce6-8f08-858104de58dc-kube-api-access-6gb98\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.054586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.050069 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af021843-9c8f-4ce6-8f08-858104de58dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.054586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.053796 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.060975 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.060951 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.072036 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.072010 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gb98\" (UniqueName: \"kubernetes.io/projected/af021843-9c8f-4ce6-8f08-858104de58dc-kube-api-access-6gb98\") pod \"maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.085433 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.085395 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl"] Oct 30 06:20:45.106341 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.106306 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" event={"ID":"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5","Type":"ContainerStarted","Data":"844864be4076d17d4e36796c10fbf86291379dbf793fb6706cd1c3ebe42069ac"} Oct 30 06:20:45.181062 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.181009 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:45.492748 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:45.492717 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc"] Oct 30 06:20:45.494285 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:20:45.494255 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf021843_9c8f_4ce6_8f08_858104de58dc.slice/crio-b94d1f845ea3673e22ff572d2ec5f2d9d4f8a21ef6c21f00d0b6450068669ef5 WatchSource:0}: Error finding container b94d1f845ea3673e22ff572d2ec5f2d9d4f8a21ef6c21f00d0b6450068669ef5: Status 404 returned error can't find the container with id b94d1f845ea3673e22ff572d2ec5f2d9d4f8a21ef6c21f00d0b6450068669ef5 Oct 30 06:20:46.117832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:46.117798 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" event={"ID":"af021843-9c8f-4ce6-8f08-858104de58dc","Type":"ContainerStarted","Data":"b94d1f845ea3673e22ff572d2ec5f2d9d4f8a21ef6c21f00d0b6450068669ef5"} Oct 30 06:20:46.119826 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:46.119796 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" event={"ID":"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5","Type":"ContainerStarted","Data":"08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4"} Oct 30 06:20:46.120112 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:46.120098 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:20:46.159320 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:46.159268 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" podStartSLOduration=2.159253602 podStartE2EDuration="2.159253602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:46.157831681 +0000 UTC m=+3005.761520938" watchObservedRunningTime="2026-10-30 06:20:46.159253602 +0000 UTC m=+3005.762942858" Oct 30 06:20:47.130161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:47.130117 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" event={"ID":"af021843-9c8f-4ce6-8f08-858104de58dc","Type":"ContainerStarted","Data":"4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057"} Oct 30 06:20:47.130676 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:47.130198 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:47.171590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:47.171529 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" podStartSLOduration=2.735814177 podStartE2EDuration="3.17151212s" totalImagesPullingTime="435.697943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:47.169019086 +0000 UTC m=+3006.772708345" watchObservedRunningTime="2026-10-30 06:20:47.17151212 +0000 UTC m=+3006.775201378" Oct 30 06:20:47.886004 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:47.885968 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc"] Oct 30 06:20:47.886309 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:47.886246 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" podUID="7ec25d1d-6e7e-4142-89c6-9a2391877e82" containerName="maas-api" containerID="cri-o://1240e3948d5a672917ce94d8bcb1e926a0a86e224104222aa2ef285dfabe838b" gracePeriod=30 Oct 30 06:20:48.464484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.463907 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" event={"ID":"7ec25d1d-6e7e-4142-89c6-9a2391877e82","Type":"ContainerDied","Data":"1240e3948d5a672917ce94d8bcb1e926a0a86e224104222aa2ef285dfabe838b"} Oct 30 06:20:48.480743 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.463782 5072 generic.go:356] "Generic (PLEG): container finished" podID="7ec25d1d-6e7e-4142-89c6-9a2391877e82" containerID="1240e3948d5a672917ce94d8bcb1e926a0a86e224104222aa2ef285dfabe838b" exitCode=0 Oct 30 06:20:48.491534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.489171 5072 generic.go:356] "Generic (PLEG): container finished" podID="fc8c667c-d266-42c4-a528-c8957316da9c" containerID="43437b170d9900a2d5f67edd8a8ea89bdaed43cddf1a7a73f4a2e2820970c3bb" exitCode=0 Oct 30 06:20:48.507515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.497124 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" event={"ID":"fc8c667c-d266-42c4-a528-c8957316da9c","Type":"ContainerDied","Data":"43437b170d9900a2d5f67edd8a8ea89bdaed43cddf1a7a73f4a2e2820970c3bb"} Oct 30 06:20:48.507515 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.500960 5072 scope.go:122] "RemoveContainer" containerID="dfce374ad48b8e35a7e37e6b884c597910886ede3a60d79bebcfd414a61707be" Oct 30 06:20:48.579879 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.579852 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:20:48.586249 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.586226 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:20:48.586978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.586962 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:20:48.587234 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.587216 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 06:20:48.666528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666424 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec25d1d-6e7e-4142-89c6-9a2391877e82-kube-api-access-x5fl2\" (UniqueName: \"kubernetes.io/projected/7ec25d1d-6e7e-4142-89c6-9a2391877e82-kube-api-access-x5fl2\") pod \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " Oct 30 06:20:48.666528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666496 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-envoy\") pod \"fc8c667c-d266-42c4-a528-c8957316da9c\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " Oct 30 06:20:48.666876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666536 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-credential-socket\") pod \"fc8c667c-d266-42c4-a528-c8957316da9c\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " Oct 30 06:20:48.666876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666576 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-certs\") pod \"fc8c667c-d266-42c4-a528-c8957316da9c\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " Oct 30 06:20:48.666876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666643 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-socket\") pod \"fc8c667c-d266-42c4-a528-c8957316da9c\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " Oct 30 06:20:48.666876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666704 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8c667c-d266-42c4-a528-c8957316da9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c667c-d266-42c4-a528-c8957316da9c-istiod-ca-cert\") pod \"fc8c667c-d266-42c4-a528-c8957316da9c\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " Oct 30 06:20:48.666876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666767 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-metrics-certs\") pod \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " Oct 30 06:20:48.667884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666881 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-data\") pod \"fc8c667c-d266-42c4-a528-c8957316da9c\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " Oct 30 06:20:48.667884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666912 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec25d1d-6e7e-4142-89c6-9a2391877e82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec25d1d-6e7e-4142-89c6-9a2391877e82-openshift-service-ca\") pod \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " Oct 30 06:20:48.667884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.666989 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-kube-api-access-xwdsr\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-kube-api-access-xwdsr\") pod \"fc8c667c-d266-42c4-a528-c8957316da9c\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " Oct 30 06:20:48.667884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.667117 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc8c667c-d266-42c4-a528-c8957316da9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8c667c-d266-42c4-a528-c8957316da9c-istio-podinfo\") pod \"fc8c667c-d266-42c4-a528-c8957316da9c\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " Oct 30 06:20:48.667884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.667159 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-maas-api-tls\") pod \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\" (UID: \"7ec25d1d-6e7e-4142-89c6-9a2391877e82\") " Oct 30 06:20:48.667884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.667189 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-istio-token\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-istio-token\") pod \"fc8c667c-d266-42c4-a528-c8957316da9c\" (UID: \"fc8c667c-d266-42c4-a528-c8957316da9c\") " Oct 30 06:20:48.668522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.668444 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-socket" pod "fc8c667c-d266-42c4-a528-c8957316da9c" (UID: "fc8c667c-d266-42c4-a528-c8957316da9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.668871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.668667 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-certs" pod "fc8c667c-d266-42c4-a528-c8957316da9c" (UID: "fc8c667c-d266-42c4-a528-c8957316da9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.669475 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.669057 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-data" pod "fc8c667c-d266-42c4-a528-c8957316da9c" (UID: "fc8c667c-d266-42c4-a528-c8957316da9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.669475 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.669215 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec25d1d-6e7e-4142-89c6-9a2391877e82-openshift-service-ca" pod "7ec25d1d-6e7e-4142-89c6-9a2391877e82" (UID: "7ec25d1d-6e7e-4142-89c6-9a2391877e82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:48.670246 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.670220 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-credential-socket" pod "fc8c667c-d266-42c4-a528-c8957316da9c" (UID: "fc8c667c-d266-42c4-a528-c8957316da9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.670411 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.670389 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8c667c-d266-42c4-a528-c8957316da9c-istiod-ca-cert" pod "fc8c667c-d266-42c4-a528-c8957316da9c" (UID: "fc8c667c-d266-42c4-a528-c8957316da9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:48.671131 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.671092 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.671131 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.671122 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec25d1d-6e7e-4142-89c6-9a2391877e82-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.671304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.671137 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.671304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.671150 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.671304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.671164 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.671304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.671178 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c667c-d266-42c4-a528-c8957316da9c-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.673075 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.673046 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-kube-api-access-xwdsr" pod "fc8c667c-d266-42c4-a528-c8957316da9c" (UID: "fc8c667c-d266-42c4-a528-c8957316da9c"). InnerVolumeSpecName "kube-api-access-xwdsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:48.675313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.675285 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec25d1d-6e7e-4142-89c6-9a2391877e82-kube-api-access-x5fl2" pod "7ec25d1d-6e7e-4142-89c6-9a2391877e82" (UID: "7ec25d1d-6e7e-4142-89c6-9a2391877e82"). InnerVolumeSpecName "kube-api-access-x5fl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:48.675424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.675378 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-envoy" pod "fc8c667c-d266-42c4-a528-c8957316da9c" (UID: "fc8c667c-d266-42c4-a528-c8957316da9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.675424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.675386 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-metrics-certs" pod "7ec25d1d-6e7e-4142-89c6-9a2391877e82" (UID: "7ec25d1d-6e7e-4142-89c6-9a2391877e82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:48.675559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.675509 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-istio-token" pod "fc8c667c-d266-42c4-a528-c8957316da9c" (UID: "fc8c667c-d266-42c4-a528-c8957316da9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:48.675620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.675593 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-maas-api-tls" pod "7ec25d1d-6e7e-4142-89c6-9a2391877e82" (UID: "7ec25d1d-6e7e-4142-89c6-9a2391877e82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:48.675726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.675709 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc8c667c-d266-42c4-a528-c8957316da9c-istio-podinfo" pod "fc8c667c-d266-42c4-a528-c8957316da9c" (UID: "fc8c667c-d266-42c4-a528-c8957316da9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:48.772313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.772228 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwdsr\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-kube-api-access-xwdsr\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.772313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.772261 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8c667c-d266-42c4-a528-c8957316da9c-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.772313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.772272 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.772313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.772280 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8c667c-d266-42c4-a528-c8957316da9c-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.772313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.772288 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5fl2\" (UniqueName: \"kubernetes.io/projected/7ec25d1d-6e7e-4142-89c6-9a2391877e82-kube-api-access-x5fl2\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.772313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.772296 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8c667c-d266-42c4-a528-c8957316da9c-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.772313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:48.772305 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec25d1d-6e7e-4142-89c6-9a2391877e82-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.348831 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.348770 5072 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:20:49.349033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.348857 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:49.499717 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.499686 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" Oct 30 06:20:49.500242 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.499684 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" event={"ID":"7ec25d1d-6e7e-4142-89c6-9a2391877e82","Type":"ContainerDied","Data":"626e7ec0fc021c3cc1a580c70fe67e38e30bc59cd99bf72f467c3f173eef1a60"} Oct 30 06:20:49.500242 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.499867 5072 scope.go:122] "RemoveContainer" containerID="1240e3948d5a672917ce94d8bcb1e926a0a86e224104222aa2ef285dfabe838b" Oct 30 06:20:49.502022 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.501948 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" Oct 30 06:20:49.502022 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.501979 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2" event={"ID":"fc8c667c-d266-42c4-a528-c8957316da9c","Type":"ContainerDied","Data":"7d53ea68a210ee122c04036fba35d8283a073403c2700cbf1b8f3a45bb10d77b"} Oct 30 06:20:49.514879 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.514859 5072 scope.go:122] "RemoveContainer" containerID="43437b170d9900a2d5f67edd8a8ea89bdaed43cddf1a7a73f4a2e2820970c3bb" Oct 30 06:20:49.551692 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.551656 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc"] Oct 30 06:20:49.554732 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.554704 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc"] Oct 30 06:20:49.599933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.599853 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2"] Oct 30 06:20:49.603564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:49.603544 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-274q2"] Oct 30 06:20:50.573649 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:50.571811 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec25d1d-6e7e-4142-89c6-9a2391877e82" path="/var/lib/kubelet/pods/7ec25d1d-6e7e-4142-89c6-9a2391877e82/volumes" Oct 30 06:20:50.602485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:50.601730 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" path="/var/lib/kubelet/pods/fc8c667c-d266-42c4-a528-c8957316da9c/volumes" Oct 30 06:20:51.465654 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:51.465610 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-1b4b7a-6cfccc5d8d-s48kc" podUID="7ec25d1d-6e7e-4142-89c6-9a2391877e82" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.159:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:53.506356 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:53.506328 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:20:56.590958 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:56.590919 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz"] Oct 30 06:20:56.591511 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:56.591174 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" containerID="cri-o://e98e6f2a57fa95e5b6fc673b41cf1df3bb39622d178f8a28da5cd0fc7071849e" gracePeriod=30 Oct 30 06:20:57.133329 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:20:57.133254 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:21:00.715629 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:21:00.715586 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:21:01.860394 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.860363 5072 generic.go:356] "Generic (PLEG): container finished" podID="9375e831-d43b-41c9-af62-082c05de351e" containerID="e98e6f2a57fa95e5b6fc673b41cf1df3bb39622d178f8a28da5cd0fc7071849e" exitCode=0 Oct 30 06:21:01.860789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.860402 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" event={"ID":"9375e831-d43b-41c9-af62-082c05de351e","Type":"ContainerDied","Data":"e98e6f2a57fa95e5b6fc673b41cf1df3bb39622d178f8a28da5cd0fc7071849e"} Oct 30 06:21:01.860789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.860500 5072 scope.go:122] "RemoveContainer" containerID="3018b74b597349dba960d173f998672bf7f0ca3c1db6a79114fb5fa3c81d7046" Oct 30 06:21:01.894584 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.894566 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 06:21:01.921967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.921938 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-envoy\") pod \"9375e831-d43b-41c9-af62-082c05de351e\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " Oct 30 06:21:01.922107 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.921982 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-socket\") pod \"9375e831-d43b-41c9-af62-082c05de351e\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " Oct 30 06:21:01.922107 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922033 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-kube-api-access-kzxbn\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-kube-api-access-kzxbn\") pod \"9375e831-d43b-41c9-af62-082c05de351e\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " Oct 30 06:21:01.922232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922175 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-credential-socket\") pod \"9375e831-d43b-41c9-af62-082c05de351e\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " Oct 30 06:21:01.922232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922221 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-istio-token\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-istio-token\") pod \"9375e831-d43b-41c9-af62-082c05de351e\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " Oct 30 06:21:01.922344 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922261 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-data\") pod \"9375e831-d43b-41c9-af62-082c05de351e\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " Oct 30 06:21:01.922344 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922290 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9375e831-d43b-41c9-af62-082c05de351e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9375e831-d43b-41c9-af62-082c05de351e-istio-podinfo\") pod \"9375e831-d43b-41c9-af62-082c05de351e\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " Oct 30 06:21:01.922344 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922294 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-socket" pod "9375e831-d43b-41c9-af62-082c05de351e" (UID: "9375e831-d43b-41c9-af62-082c05de351e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:01.922344 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922325 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-certs\") pod \"9375e831-d43b-41c9-af62-082c05de351e\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " Oct 30 06:21:01.922609 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922367 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9375e831-d43b-41c9-af62-082c05de351e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9375e831-d43b-41c9-af62-082c05de351e-istiod-ca-cert\") pod \"9375e831-d43b-41c9-af62-082c05de351e\" (UID: \"9375e831-d43b-41c9-af62-082c05de351e\") " Oct 30 06:21:01.922609 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922439 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-credential-socket" pod "9375e831-d43b-41c9-af62-082c05de351e" (UID: "9375e831-d43b-41c9-af62-082c05de351e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:01.922609 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922536 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-data" pod "9375e831-d43b-41c9-af62-082c05de351e" (UID: "9375e831-d43b-41c9-af62-082c05de351e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:01.922865 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922839 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-certs" pod "9375e831-d43b-41c9-af62-082c05de351e" (UID: "9375e831-d43b-41c9-af62-082c05de351e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:01.922936 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.922920 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9375e831-d43b-41c9-af62-082c05de351e-istiod-ca-cert" pod "9375e831-d43b-41c9-af62-082c05de351e" (UID: "9375e831-d43b-41c9-af62-082c05de351e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:01.923159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.923139 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:01.923252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.923163 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:01.923252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.923176 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:01.923252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.923192 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:01.923252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.923204 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9375e831-d43b-41c9-af62-082c05de351e-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:01.925332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.925302 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-istio-token" pod "9375e831-d43b-41c9-af62-082c05de351e" (UID: "9375e831-d43b-41c9-af62-082c05de351e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:01.925444 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.925339 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9375e831-d43b-41c9-af62-082c05de351e-istio-podinfo" pod "9375e831-d43b-41c9-af62-082c05de351e" (UID: "9375e831-d43b-41c9-af62-082c05de351e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:01.926005 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.925972 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-envoy" pod "9375e831-d43b-41c9-af62-082c05de351e" (UID: "9375e831-d43b-41c9-af62-082c05de351e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:01.926374 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:01.926352 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-kube-api-access-kzxbn" pod "9375e831-d43b-41c9-af62-082c05de351e" (UID: "9375e831-d43b-41c9-af62-082c05de351e"). InnerVolumeSpecName "kube-api-access-kzxbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:02.024161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:02.024123 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9375e831-d43b-41c9-af62-082c05de351e-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.024161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:02.024163 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxbn\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-kube-api-access-kzxbn\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.024389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:02.024175 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9375e831-d43b-41c9-af62-082c05de351e-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.024389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:02.024183 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9375e831-d43b-41c9-af62-082c05de351e-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.871508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:02.871471 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" event={"ID":"9375e831-d43b-41c9-af62-082c05de351e","Type":"ContainerDied","Data":"dadcc0f5fc74718d1b69c972cc8239b6cedb9e4c557be01ac2ea129a1ebd10a6"} Oct 30 06:21:02.871508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:02.871505 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz" Oct 30 06:21:02.872025 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:02.871578 5072 scope.go:122] "RemoveContainer" containerID="e98e6f2a57fa95e5b6fc673b41cf1df3bb39622d178f8a28da5cd0fc7071849e" Oct 30 06:21:02.963358 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:02.963308 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz"] Oct 30 06:21:02.971110 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:02.971073 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-kx8gz"] Oct 30 06:21:04.520397 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:04.520357 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9375e831-d43b-41c9-af62-082c05de351e" path="/var/lib/kubelet/pods/9375e831-d43b-41c9-af62-082c05de351e/volumes" Oct 30 06:21:04.939815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:04.939783 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl_2b0d06fe-f98e-4c1b-8669-aa8ee23164f5/payload-processing/0.log" Oct 30 06:21:19.526687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:19.526636 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:21:19.527389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:19.526728 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:21:22.037880 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:22.037796 5072 generic.go:356] "Generic (PLEG): container finished" podID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerID="6ca4efaf2d9c50e7b0393d48c3803f210bc380f94e8375273aea3246fc41d33a" exitCode=0 Oct 30 06:21:22.038339 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:22.037871 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" event={"ID":"3bf8ac2d-724c-4309-aaba-de0952fc9802","Type":"ContainerDied","Data":"6ca4efaf2d9c50e7b0393d48c3803f210bc380f94e8375273aea3246fc41d33a"} Oct 30 06:21:22.038339 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:22.038329 5072 scope.go:122] "RemoveContainer" containerID="6ca4efaf2d9c50e7b0393d48c3803f210bc380f94e8375273aea3246fc41d33a" Oct 30 06:21:22.040604 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:22.040570 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:21:22.040685 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:22.040665 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:21:22.040731 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:22.040717 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:21:23.047399 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:23.047363 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" event={"ID":"3bf8ac2d-724c-4309-aaba-de0952fc9802","Type":"ContainerStarted","Data":"db633516ed6556654a8babcaa65d56636d58e4a4e7f64950b8aba02bb6e73a43"} Oct 30 06:21:23.469726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:23.469689 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:21:23.469970 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:23.469947 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:21:23.470065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:23.469999 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:21:24.470123 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:24.470082 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:21:24.470569 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:24.470145 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:21:25.470513 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:25.470477 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:21:25.470911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:25.470529 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:21:26.470248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:26.470213 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:21:26.470429 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:26.470274 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:21:27.443632 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:27.443592 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665"] Oct 30 06:21:27.444173 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:27.443893 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" containerID="cri-o://8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8" gracePeriod=30 Oct 30 06:21:27.472542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:27.471237 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:21:27.472542 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:27.471306 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 06:21:28.743078 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:28.743026 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:21:28.744024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:28.743950 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:21:28.745096 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:28.745070 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:21:30.711922 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:21:30.711870 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:21:32.817445 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.817418 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:21:32.939454 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.939419 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-envoy\") pod \"81823e59-03bb-42fc-ac26-2ff5b93d3898\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " Oct 30 06:21:32.939454 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.939468 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-socket\") pod \"81823e59-03bb-42fc-ac26-2ff5b93d3898\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " Oct 30 06:21:32.939777 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.939623 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81823e59-03bb-42fc-ac26-2ff5b93d3898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81823e59-03bb-42fc-ac26-2ff5b93d3898-istiod-ca-cert\") pod \"81823e59-03bb-42fc-ac26-2ff5b93d3898\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " Oct 30 06:21:32.939777 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.939660 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-token\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-token\") pod \"81823e59-03bb-42fc-ac26-2ff5b93d3898\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " Oct 30 06:21:32.939925 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.939784 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-kube-api-access-ns7lf\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-kube-api-access-ns7lf\") pod \"81823e59-03bb-42fc-ac26-2ff5b93d3898\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " Oct 30 06:21:32.939925 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.939891 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-podinfo\") pod \"81823e59-03bb-42fc-ac26-2ff5b93d3898\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " Oct 30 06:21:32.940031 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.939931 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-data\") pod \"81823e59-03bb-42fc-ac26-2ff5b93d3898\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " Oct 30 06:21:32.940031 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.939970 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-credential-socket\") pod \"81823e59-03bb-42fc-ac26-2ff5b93d3898\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " Oct 30 06:21:32.940135 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.940049 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-certs\") pod \"81823e59-03bb-42fc-ac26-2ff5b93d3898\" (UID: \"81823e59-03bb-42fc-ac26-2ff5b93d3898\") " Oct 30 06:21:32.940206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.940146 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81823e59-03bb-42fc-ac26-2ff5b93d3898-istiod-ca-cert" pod "81823e59-03bb-42fc-ac26-2ff5b93d3898" (UID: "81823e59-03bb-42fc-ac26-2ff5b93d3898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:32.941148 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.941119 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81823e59-03bb-42fc-ac26-2ff5b93d3898-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:32.941384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.941357 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-certs" pod "81823e59-03bb-42fc-ac26-2ff5b93d3898" (UID: "81823e59-03bb-42fc-ac26-2ff5b93d3898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:32.941595 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.941546 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-data" pod "81823e59-03bb-42fc-ac26-2ff5b93d3898" (UID: "81823e59-03bb-42fc-ac26-2ff5b93d3898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:32.944550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.941798 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-credential-socket" pod "81823e59-03bb-42fc-ac26-2ff5b93d3898" (UID: "81823e59-03bb-42fc-ac26-2ff5b93d3898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:32.944550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.942924 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-podinfo" pod "81823e59-03bb-42fc-ac26-2ff5b93d3898" (UID: "81823e59-03bb-42fc-ac26-2ff5b93d3898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:32.944550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.943886 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-envoy" pod "81823e59-03bb-42fc-ac26-2ff5b93d3898" (UID: "81823e59-03bb-42fc-ac26-2ff5b93d3898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:32.947299 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.947254 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-token" pod "81823e59-03bb-42fc-ac26-2ff5b93d3898" (UID: "81823e59-03bb-42fc-ac26-2ff5b93d3898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:32.952101 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.952073 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-kube-api-access-ns7lf" pod "81823e59-03bb-42fc-ac26-2ff5b93d3898" (UID: "81823e59-03bb-42fc-ac26-2ff5b93d3898"). InnerVolumeSpecName "kube-api-access-ns7lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:32.953122 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:32.953098 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-socket" pod "81823e59-03bb-42fc-ac26-2ff5b93d3898" (UID: "81823e59-03bb-42fc-ac26-2ff5b93d3898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:33.042980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.042886 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.042980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.042915 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.042980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.042924 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.042980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.042932 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.042980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.042941 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns7lf\" (UniqueName: \"kubernetes.io/projected/81823e59-03bb-42fc-ac26-2ff5b93d3898-kube-api-access-ns7lf\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.042980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.042952 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.042980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.042964 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.042980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.042978 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81823e59-03bb-42fc-ac26-2ff5b93d3898-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.126096 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.126059 5072 generic.go:356] "Generic (PLEG): container finished" podID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerID="8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8" exitCode=0 Oct 30 06:21:33.126308 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.126132 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" Oct 30 06:21:33.126308 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.126149 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" event={"ID":"81823e59-03bb-42fc-ac26-2ff5b93d3898","Type":"ContainerDied","Data":"8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8"} Oct 30 06:21:33.126308 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.126208 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665" event={"ID":"81823e59-03bb-42fc-ac26-2ff5b93d3898","Type":"ContainerDied","Data":"af594574e69e36894bf93ba1b7cdaf7862417b5f85e31b8b28793f6f87c3dbf1"} Oct 30 06:21:33.126308 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.126241 5072 scope.go:122] "RemoveContainer" containerID="8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8" Oct 30 06:21:33.138478 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.138438 5072 scope.go:122] "RemoveContainer" containerID="8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8" Oct 30 06:21:33.138836 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:21:33.138814 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8\": container with ID starting with 8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8 not found: ID does not exist" containerID="8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8" Oct 30 06:21:33.138935 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.138848 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8"} err="failed to get container status \"8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8\": rpc error: code = NotFound desc = could not find container \"8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8\": container with ID starting with 8c0735d83957ac1303953f1150e66637967e3a29229b7321b5c835ca2abb66c8 not found: ID does not exist" Oct 30 06:21:33.176352 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.176322 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665"] Oct 30 06:21:33.188059 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:33.188022 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-df665"] Oct 30 06:21:34.519872 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:34.519840 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" path="/var/lib/kubelet/pods/81823e59-03bb-42fc-ac26-2ff5b93d3898/volumes" Oct 30 06:21:35.463642 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:35.463607 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds"] Oct 30 06:21:35.464421 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:35.463931 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" podUID="a8b1bd51-0d34-4d80-b0f7-80024906a8a5" containerName="maas-api" containerID="cri-o://c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252" gracePeriod=30 Oct 30 06:21:35.993483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:35.993100 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:21:36.122695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.122666 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-maas-api-tls\") pod \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " Oct 30 06:21:36.122920 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.122799 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-openshift-service-ca\") pod \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " Oct 30 06:21:36.122920 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.122839 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-metrics-certs\") pod \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " Oct 30 06:21:36.123109 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.122932 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-kube-api-access-c6pcc\" (UniqueName: \"kubernetes.io/projected/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-kube-api-access-c6pcc\") pod \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\" (UID: \"a8b1bd51-0d34-4d80-b0f7-80024906a8a5\") " Oct 30 06:21:36.124526 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.124496 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-openshift-service-ca" pod "a8b1bd51-0d34-4d80-b0f7-80024906a8a5" (UID: "a8b1bd51-0d34-4d80-b0f7-80024906a8a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:36.126794 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.126422 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-maas-api-tls" pod "a8b1bd51-0d34-4d80-b0f7-80024906a8a5" (UID: "a8b1bd51-0d34-4d80-b0f7-80024906a8a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:36.127178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.127152 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-kube-api-access-c6pcc" pod "a8b1bd51-0d34-4d80-b0f7-80024906a8a5" (UID: "a8b1bd51-0d34-4d80-b0f7-80024906a8a5"). InnerVolumeSpecName "kube-api-access-c6pcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:36.127840 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.127705 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-metrics-certs" pod "a8b1bd51-0d34-4d80-b0f7-80024906a8a5" (UID: "a8b1bd51-0d34-4d80-b0f7-80024906a8a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:36.173002 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.172906 5072 generic.go:356] "Generic (PLEG): container finished" podID="a8b1bd51-0d34-4d80-b0f7-80024906a8a5" containerID="c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252" exitCode=0 Oct 30 06:21:36.173002 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.172970 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" Oct 30 06:21:36.173002 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.172986 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" event={"ID":"a8b1bd51-0d34-4d80-b0f7-80024906a8a5","Type":"ContainerDied","Data":"c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252"} Oct 30 06:21:36.173305 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.173037 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds" event={"ID":"a8b1bd51-0d34-4d80-b0f7-80024906a8a5","Type":"ContainerDied","Data":"68a0a27718c651b5a545632a9d63726f95f4ff9b4bd1f420e31dd1df421d535d"} Oct 30 06:21:36.173305 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.173060 5072 scope.go:122] "RemoveContainer" containerID="c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252" Oct 30 06:21:36.188073 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.188050 5072 scope.go:122] "RemoveContainer" containerID="c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252" Oct 30 06:21:36.188534 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:21:36.188511 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252\": container with ID starting with c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252 not found: ID does not exist" containerID="c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252" Oct 30 06:21:36.188650 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.188540 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252"} err="failed to get container status \"c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252\": rpc error: code = NotFound desc = could not find container \"c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252\": container with ID starting with c7be8254ca0f9bfe5658cc180440bfbc635ee9f1fbbb365754bfb65cae760252 not found: ID does not exist" Oct 30 06:21:36.225031 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.224993 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:36.225031 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.225025 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:36.225212 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.225039 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6pcc\" (UniqueName: \"kubernetes.io/projected/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-kube-api-access-c6pcc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:36.225212 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.225051 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b1bd51-0d34-4d80-b0f7-80024906a8a5-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:36.225303 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.225283 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds"] Oct 30 06:21:36.230713 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.230691 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-02ceed-5fc6f6db6c-4sxds"] Oct 30 06:21:36.523493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:36.523447 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b1bd51-0d34-4d80-b0f7-80024906a8a5" path="/var/lib/kubelet/pods/a8b1bd51-0d34-4d80-b0f7-80024906a8a5/volumes" Oct 30 06:21:43.649837 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:43.649801 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t"] Oct 30 06:21:43.650425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:43.650073 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" podUID="9a517262-378e-400c-817d-3b72b61ac3ee" containerName="maas-api" containerID="cri-o://77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181" gracePeriod=30 Oct 30 06:21:43.836824 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:43.836403 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr"] Oct 30 06:21:43.836824 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:43.836688 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" podUID="73d2ac8a-7330-4daa-88f3-2e6db8313230" containerName="payload-pre-processing" containerID="cri-o://817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44" gracePeriod=30 Oct 30 06:21:44.020244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.019578 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:21:44.148074 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.148048 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a517262-378e-400c-817d-3b72b61ac3ee-kube-api-access-mvkzh\" (UniqueName: \"kubernetes.io/projected/9a517262-378e-400c-817d-3b72b61ac3ee-kube-api-access-mvkzh\") pod \"9a517262-378e-400c-817d-3b72b61ac3ee\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " Oct 30 06:21:44.148241 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.148082 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-metrics-certs\") pod \"9a517262-378e-400c-817d-3b72b61ac3ee\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " Oct 30 06:21:44.148241 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.148223 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a517262-378e-400c-817d-3b72b61ac3ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a517262-378e-400c-817d-3b72b61ac3ee-openshift-service-ca\") pod \"9a517262-378e-400c-817d-3b72b61ac3ee\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " Oct 30 06:21:44.148602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.148571 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a517262-378e-400c-817d-3b72b61ac3ee-openshift-service-ca" pod "9a517262-378e-400c-817d-3b72b61ac3ee" (UID: "9a517262-378e-400c-817d-3b72b61ac3ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:44.148767 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.148751 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls\") pod \"9a517262-378e-400c-817d-3b72b61ac3ee\" (UID: \"9a517262-378e-400c-817d-3b72b61ac3ee\") " Oct 30 06:21:44.149693 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.149668 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a517262-378e-400c-817d-3b72b61ac3ee-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.151206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.151106 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls" pod "9a517262-378e-400c-817d-3b72b61ac3ee" (UID: "9a517262-378e-400c-817d-3b72b61ac3ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:44.151206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.151182 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-metrics-certs" pod "9a517262-378e-400c-817d-3b72b61ac3ee" (UID: "9a517262-378e-400c-817d-3b72b61ac3ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:44.152012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.151991 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a517262-378e-400c-817d-3b72b61ac3ee-kube-api-access-mvkzh" pod "9a517262-378e-400c-817d-3b72b61ac3ee" (UID: "9a517262-378e-400c-817d-3b72b61ac3ee"). InnerVolumeSpecName "kube-api-access-mvkzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:44.172433 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.172403 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.284111 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d2ac8a-7330-4daa-88f3-2e6db8313230-kube-api-access-vngzh\" (UniqueName: \"kubernetes.io/projected/73d2ac8a-7330-4daa-88f3-2e6db8313230-kube-api-access-vngzh\") pod \"73d2ac8a-7330-4daa-88f3-2e6db8313230\" (UID: \"73d2ac8a-7330-4daa-88f3-2e6db8313230\") " Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.284223 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d2ac8a-7330-4daa-88f3-2e6db8313230-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d2ac8a-7330-4daa-88f3-2e6db8313230-plugins-config-volume\") pod \"73d2ac8a-7330-4daa-88f3-2e6db8313230\" (UID: \"73d2ac8a-7330-4daa-88f3-2e6db8313230\") " Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.284677 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvkzh\" (UniqueName: \"kubernetes.io/projected/9a517262-378e-400c-817d-3b72b61ac3ee-kube-api-access-mvkzh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.284697 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.284711 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a517262-378e-400c-817d-3b72b61ac3ee-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.335207 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d2ac8a-7330-4daa-88f3-2e6db8313230-kube-api-access-vngzh" pod "73d2ac8a-7330-4daa-88f3-2e6db8313230" (UID: "73d2ac8a-7330-4daa-88f3-2e6db8313230"). InnerVolumeSpecName "kube-api-access-vngzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.349000 5072 generic.go:356] "Generic (PLEG): container finished" podID="73d2ac8a-7330-4daa-88f3-2e6db8313230" containerID="817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44" exitCode=0 Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.349229 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" event={"ID":"73d2ac8a-7330-4daa-88f3-2e6db8313230","Type":"ContainerDied","Data":"817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44"} Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.349282 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" event={"ID":"73d2ac8a-7330-4daa-88f3-2e6db8313230","Type":"ContainerDied","Data":"b1308df0d8808972bc060c9b31289e9c2f5e7ef80b53db167fe4b089ae9b4dc3"} Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.349312 5072 scope.go:122] "RemoveContainer" containerID="817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44" Oct 30 06:21:44.352414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.349448 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr" Oct 30 06:21:44.353719 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.353661 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d2ac8a-7330-4daa-88f3-2e6db8313230-plugins-config-volume" pod "73d2ac8a-7330-4daa-88f3-2e6db8313230" (UID: "73d2ac8a-7330-4daa-88f3-2e6db8313230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:44.355595 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.353954 5072 generic.go:356] "Generic (PLEG): container finished" podID="9a517262-378e-400c-817d-3b72b61ac3ee" containerID="77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181" exitCode=0 Oct 30 06:21:44.355595 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.354005 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" Oct 30 06:21:44.355595 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.354048 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" event={"ID":"9a517262-378e-400c-817d-3b72b61ac3ee","Type":"ContainerDied","Data":"77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181"} Oct 30 06:21:44.355595 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.354087 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t" event={"ID":"9a517262-378e-400c-817d-3b72b61ac3ee","Type":"ContainerDied","Data":"e09bb26bb0fdcdd47df3a7e828c98a0eb46b42279fde646cb4f1353aec65bd76"} Oct 30 06:21:44.386588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.386555 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vngzh\" (UniqueName: \"kubernetes.io/projected/73d2ac8a-7330-4daa-88f3-2e6db8313230-kube-api-access-vngzh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.386588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.386588 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d2ac8a-7330-4daa-88f3-2e6db8313230-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.388763 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.388745 5072 scope.go:122] "RemoveContainer" containerID="817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44" Oct 30 06:21:44.389475 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:21:44.389436 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44\": container with ID starting with 817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44 not found: ID does not exist" containerID="817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44" Oct 30 06:21:44.389640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.389611 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44"} err="failed to get container status \"817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44\": rpc error: code = NotFound desc = could not find container \"817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44\": container with ID starting with 817e59b8b8667008dc6b63bf64ec031e870ccc2b1156f705cb3b5dc4b07bed44 not found: ID does not exist" Oct 30 06:21:44.389750 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.389730 5072 scope.go:122] "RemoveContainer" containerID="77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181" Oct 30 06:21:44.406915 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.406895 5072 scope.go:122] "RemoveContainer" containerID="77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181" Oct 30 06:21:44.407242 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:21:44.407212 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181\": container with ID starting with 77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181 not found: ID does not exist" containerID="77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181" Oct 30 06:21:44.407344 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.407249 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181"} err="failed to get container status \"77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181\": rpc error: code = NotFound desc = could not find container \"77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181\": container with ID starting with 77f28a7f4e27e2442a1e0e7311c9ef1bb6cfdf821ce150821e60e7d7cc709181 not found: ID does not exist" Oct 30 06:21:44.423534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.423487 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t"] Oct 30 06:21:44.427087 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.427061 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fb9345-6689d69c54-8l95t"] Oct 30 06:21:44.519440 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.519410 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a517262-378e-400c-817d-3b72b61ac3ee" path="/var/lib/kubelet/pods/9a517262-378e-400c-817d-3b72b61ac3ee/volumes" Oct 30 06:21:44.718310 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.718271 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr"] Oct 30 06:21:44.723995 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:44.723966 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb9345-d8b8c76bc-66txr"] Oct 30 06:21:46.527862 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:46.524630 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d2ac8a-7330-4daa-88f3-2e6db8313230" path="/var/lib/kubelet/pods/73d2ac8a-7330-4daa-88f3-2e6db8313230/volumes" Oct 30 06:21:49.425586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:49.425550 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj"] Oct 30 06:21:49.436321 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:49.436288 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b"] Oct 30 06:21:51.617667 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:51.617613 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc"] Oct 30 06:21:51.618214 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:51.617863 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" containerID="cri-o://06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d" gracePeriod=30 Oct 30 06:21:56.915026 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.914995 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:21:56.996001 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.995964 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-certs\") pod \"854b69be-96e5-4912-8ba9-1df7ce008e7c\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " Oct 30 06:21:56.996213 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996046 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-token\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-token\") pod \"854b69be-96e5-4912-8ba9-1df7ce008e7c\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " Oct 30 06:21:56.996213 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996109 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-socket\") pod \"854b69be-96e5-4912-8ba9-1df7ce008e7c\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " Oct 30 06:21:56.996213 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996150 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-credential-socket\") pod \"854b69be-96e5-4912-8ba9-1df7ce008e7c\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " Oct 30 06:21:56.996510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996214 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-podinfo\") pod \"854b69be-96e5-4912-8ba9-1df7ce008e7c\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " Oct 30 06:21:56.996510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996276 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-envoy\") pod \"854b69be-96e5-4912-8ba9-1df7ce008e7c\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " Oct 30 06:21:56.996510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996275 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-certs" pod "854b69be-96e5-4912-8ba9-1df7ce008e7c" (UID: "854b69be-96e5-4912-8ba9-1df7ce008e7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:56.996510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996392 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854b69be-96e5-4912-8ba9-1df7ce008e7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854b69be-96e5-4912-8ba9-1df7ce008e7c-istiod-ca-cert\") pod \"854b69be-96e5-4912-8ba9-1df7ce008e7c\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " Oct 30 06:21:56.996510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996495 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-data\") pod \"854b69be-96e5-4912-8ba9-1df7ce008e7c\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " Oct 30 06:21:56.996510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996500 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-socket" pod "854b69be-96e5-4912-8ba9-1df7ce008e7c" (UID: "854b69be-96e5-4912-8ba9-1df7ce008e7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:56.996842 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.996626 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-kube-api-access-mlm4z\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-kube-api-access-mlm4z\") pod \"854b69be-96e5-4912-8ba9-1df7ce008e7c\" (UID: \"854b69be-96e5-4912-8ba9-1df7ce008e7c\") " Oct 30 06:21:56.997250 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.997223 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:56.997371 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.997253 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:56.999904 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:56.999718 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-envoy" pod "854b69be-96e5-4912-8ba9-1df7ce008e7c" (UID: "854b69be-96e5-4912-8ba9-1df7ce008e7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:57.000128 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.000003 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-kube-api-access-mlm4z" pod "854b69be-96e5-4912-8ba9-1df7ce008e7c" (UID: "854b69be-96e5-4912-8ba9-1df7ce008e7c"). InnerVolumeSpecName "kube-api-access-mlm4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:57.000128 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.000057 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-credential-socket" pod "854b69be-96e5-4912-8ba9-1df7ce008e7c" (UID: "854b69be-96e5-4912-8ba9-1df7ce008e7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:57.000528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.000341 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854b69be-96e5-4912-8ba9-1df7ce008e7c-istiod-ca-cert" pod "854b69be-96e5-4912-8ba9-1df7ce008e7c" (UID: "854b69be-96e5-4912-8ba9-1df7ce008e7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:57.000716 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.000632 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-data" pod "854b69be-96e5-4912-8ba9-1df7ce008e7c" (UID: "854b69be-96e5-4912-8ba9-1df7ce008e7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:57.004337 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.004314 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-token" pod "854b69be-96e5-4912-8ba9-1df7ce008e7c" (UID: "854b69be-96e5-4912-8ba9-1df7ce008e7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:57.004420 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.004396 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-podinfo" pod "854b69be-96e5-4912-8ba9-1df7ce008e7c" (UID: "854b69be-96e5-4912-8ba9-1df7ce008e7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:57.098262 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.098234 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.098262 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.098262 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854b69be-96e5-4912-8ba9-1df7ce008e7c-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.098505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.098272 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.098505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.098281 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlm4z\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-kube-api-access-mlm4z\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.098505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.098290 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.098505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.098297 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854b69be-96e5-4912-8ba9-1df7ce008e7c-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.098505 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.098305 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854b69be-96e5-4912-8ba9-1df7ce008e7c-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.472191 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.472150 5072 generic.go:356] "Generic (PLEG): container finished" podID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerID="06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d" exitCode=0 Oct 30 06:21:57.472407 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.472232 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" Oct 30 06:21:57.472407 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.472241 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" event={"ID":"854b69be-96e5-4912-8ba9-1df7ce008e7c","Type":"ContainerDied","Data":"06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d"} Oct 30 06:21:57.472407 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.472314 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc" event={"ID":"854b69be-96e5-4912-8ba9-1df7ce008e7c","Type":"ContainerDied","Data":"27d55b0243fc63f2dbf09ba0886cab1e9858280535a28bc550e316f5acc27623"} Oct 30 06:21:57.472407 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.472377 5072 scope.go:122] "RemoveContainer" containerID="06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d" Oct 30 06:21:57.484868 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.484846 5072 scope.go:122] "RemoveContainer" containerID="95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34" Oct 30 06:21:57.497939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.497914 5072 scope.go:122] "RemoveContainer" containerID="06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d" Oct 30 06:21:57.498229 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:21:57.498207 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d\": container with ID starting with 06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d not found: ID does not exist" containerID="06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d" Oct 30 06:21:57.498303 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.498239 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d"} err="failed to get container status \"06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d\": rpc error: code = NotFound desc = could not find container \"06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d\": container with ID starting with 06aeed82067cba80bbb3cdecb711f5ee0ba5ed3806843f455436f319be0f176d not found: ID does not exist" Oct 30 06:21:57.498303 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.498256 5072 scope.go:122] "RemoveContainer" containerID="95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34" Oct 30 06:21:57.498522 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:21:57.498505 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34\": container with ID starting with 95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34 not found: ID does not exist" containerID="95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34" Oct 30 06:21:57.498588 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.498526 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34"} err="failed to get container status \"95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34\": rpc error: code = NotFound desc = could not find container \"95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34\": container with ID starting with 95b81a46de4c65eb23c8c1319ad0a980e5ad9d590ee38cf109531e1412f96d34 not found: ID does not exist" Oct 30 06:21:57.521564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.521539 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc"] Oct 30 06:21:57.533751 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:57.533722 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fb9345-openshift-default-757c4d97fc-2xqxc"] Oct 30 06:21:58.521784 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:21:58.521752 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" path="/var/lib/kubelet/pods/854b69be-96e5-4912-8ba9-1df7ce008e7c/volumes" Oct 30 06:22:00.710415 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:00.710369 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:22:04.450662 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.450613 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" containerID="cri-o://894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb" gracePeriod=45 Oct 30 06:22:04.459639 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.459600 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" containerID="cri-o://00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490" gracePeriod=45 Oct 30 06:22:04.836197 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.836167 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:22:04.904198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.903740 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 06:22:04.973346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.973308 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tmp-dir\") pod \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " Oct 30 06:22:04.973652 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.973629 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-kube-api-access-7q6tx\" (UniqueName: \"kubernetes.io/projected/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-kube-api-access-7q6tx\") pod \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " Oct 30 06:22:04.973952 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.973935 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-dshm\") pod \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " Oct 30 06:22:04.974134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.974119 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-model-cache\") pod \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " Oct 30 06:22:04.974275 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.974260 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tls-certs\") pod \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " Oct 30 06:22:04.974444 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.974429 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-home\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-home\") pod \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\" (UID: \"9e70915b-1cdf-4270-81eb-5c7c8f9a5313\") " Oct 30 06:22:04.975707 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.975677 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-model-cache" pod "9e70915b-1cdf-4270-81eb-5c7c8f9a5313" (UID: "9e70915b-1cdf-4270-81eb-5c7c8f9a5313"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:04.975844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.975693 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-home" pod "9e70915b-1cdf-4270-81eb-5c7c8f9a5313" (UID: "9e70915b-1cdf-4270-81eb-5c7c8f9a5313"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:04.976081 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.976057 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tmp-dir" pod "9e70915b-1cdf-4270-81eb-5c7c8f9a5313" (UID: "9e70915b-1cdf-4270-81eb-5c7c8f9a5313"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:04.979284 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.978839 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-dshm" pod "9e70915b-1cdf-4270-81eb-5c7c8f9a5313" (UID: "9e70915b-1cdf-4270-81eb-5c7c8f9a5313"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:04.979284 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.978927 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tls-certs" pod "9e70915b-1cdf-4270-81eb-5c7c8f9a5313" (UID: "9e70915b-1cdf-4270-81eb-5c7c8f9a5313"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:04.981506 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.981426 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-kube-api-access-7q6tx" pod "9e70915b-1cdf-4270-81eb-5c7c8f9a5313" (UID: "9e70915b-1cdf-4270-81eb-5c7c8f9a5313"). InnerVolumeSpecName "kube-api-access-7q6tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:04.997316 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.997290 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm"] Oct 30 06:22:04.998170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998151 5072 cpu_manager.go:414] "containerMap: removing container" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" Oct 30 06:22:04.998279 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998270 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" Oct 30 06:22:04.998345 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998338 5072 cpu_manager.go:414] "containerMap: removing container" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" Oct 30 06:22:04.998396 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998389 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" Oct 30 06:22:04.998507 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998452 5072 cpu_manager.go:414] "containerMap: removing container" podUID="a8b1bd51-0d34-4d80-b0f7-80024906a8a5" containerName="maas-api" Oct 30 06:22:04.998583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998574 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b1bd51-0d34-4d80-b0f7-80024906a8a5" containerName="maas-api" Oct 30 06:22:04.998639 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998633 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" Oct 30 06:22:04.998688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998680 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" Oct 30 06:22:04.998741 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998734 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" Oct 30 06:22:04.998783 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998777 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" Oct 30 06:22:04.998835 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998829 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" Oct 30 06:22:04.998887 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998880 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" Oct 30 06:22:04.998937 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998931 5072 cpu_manager.go:414] "containerMap: removing container" podUID="7ec25d1d-6e7e-4142-89c6-9a2391877e82" containerName="maas-api" Oct 30 06:22:04.998986 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.998980 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec25d1d-6e7e-4142-89c6-9a2391877e82" containerName="maas-api" Oct 30 06:22:04.999034 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999028 5072 cpu_manager.go:414] "containerMap: removing container" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" Oct 30 06:22:04.999081 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999075 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" Oct 30 06:22:04.999134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999128 5072 cpu_manager.go:414] "containerMap: removing container" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" Oct 30 06:22:04.999186 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999180 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" Oct 30 06:22:04.999242 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999235 5072 cpu_manager.go:414] "containerMap: removing container" podUID="73d2ac8a-7330-4daa-88f3-2e6db8313230" containerName="payload-pre-processing" Oct 30 06:22:04.999289 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999283 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d2ac8a-7330-4daa-88f3-2e6db8313230" containerName="payload-pre-processing" Oct 30 06:22:04.999341 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999335 5072 cpu_manager.go:414] "containerMap: removing container" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" Oct 30 06:22:04.999388 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999381 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" Oct 30 06:22:04.999445 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999438 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9a517262-378e-400c-817d-3b72b61ac3ee" containerName="maas-api" Oct 30 06:22:04.999524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999512 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a517262-378e-400c-817d-3b72b61ac3ee" containerName="maas-api" Oct 30 06:22:04.999586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999579 5072 cpu_manager.go:414] "containerMap: removing container" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" Oct 30 06:22:04.999635 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999629 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" Oct 30 06:22:04.999904 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:04.999888 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" Oct 30 06:22:05.000019 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.000007 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d2ac8a-7330-4daa-88f3-2e6db8313230" containerName="payload-pre-processing" Oct 30 06:22:05.000784 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.000768 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="854b69be-96e5-4912-8ba9-1df7ce008e7c" containerName="istio-proxy" Oct 30 06:22:05.000912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.000901 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a517262-378e-400c-817d-3b72b61ac3ee" containerName="maas-api" Oct 30 06:22:05.001600 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.000987 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" Oct 30 06:22:05.001734 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.001708 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="81823e59-03bb-42fc-ac26-2ff5b93d3898" containerName="istio-proxy" Oct 30 06:22:05.001801 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.001791 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" Oct 30 06:22:05.001854 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.001848 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b1bd51-0d34-4d80-b0f7-80024906a8a5" containerName="maas-api" Oct 30 06:22:05.001909 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.001903 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerName="main" Oct 30 06:22:05.001997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.001986 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" Oct 30 06:22:05.002240 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.002229 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9375e831-d43b-41c9-af62-082c05de351e" containerName="istio-proxy" Oct 30 06:22:05.002316 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.002309 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec25d1d-6e7e-4142-89c6-9a2391877e82" containerName="maas-api" Oct 30 06:22:05.006079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.006063 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:05.028189 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.027101 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-718730\"" Oct 30 06:22:05.028189 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.027232 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-718730-dockercfg-rj9fd\"" Oct 30 06:22:05.029131 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.028996 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm"] Oct 30 06:22:05.086547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.079350 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tmp-dir\") pod \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " Oct 30 06:22:05.086547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.079477 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tls-certs\" (UniqueName: \"kubernetes.io/secret/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tls-certs\") pod \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " Oct 30 06:22:05.086547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.079533 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-kube-api-access-qxx26\" (UniqueName: \"kubernetes.io/projected/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-kube-api-access-qxx26\") pod \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " Oct 30 06:22:05.086547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.079580 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-dshm\") pod \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " Oct 30 06:22:05.086547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.079629 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-home\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-home\") pod \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " Oct 30 06:22:05.086547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.079781 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-model-cache\") pod \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\" (UID: \"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507\") " Oct 30 06:22:05.131483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.131239 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.131483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.131279 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.131483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.131295 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.131483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.131307 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.131483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.131320 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.131483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.131334 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q6tx\" (UniqueName: \"kubernetes.io/projected/9e70915b-1cdf-4270-81eb-5c7c8f9a5313-kube-api-access-7q6tx\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.166517 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.155837 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tmp-dir" pod "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" (UID: "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:05.166517 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.162596 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-model-cache" pod "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" (UID: "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:05.209484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.208880 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-home" pod "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" (UID: "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:05.225910 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.217500 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7"] Oct 30 06:22:05.225910 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.218744 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8c667c-d266-42c4-a528-c8957316da9c" containerName="istio-proxy" Oct 30 06:22:05.237493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.233513 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grdq\" (UniqueName: \"kubernetes.io/projected/74387953-1299-4d3b-8e25-995bd29ce315-kube-api-access-6grdq\") pod \"payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm\" (UID: \"74387953-1299-4d3b-8e25-995bd29ce315\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:05.237493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.233652 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74387953-1299-4d3b-8e25-995bd29ce315-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm\" (UID: \"74387953-1299-4d3b-8e25-995bd29ce315\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:05.237493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.234058 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.237493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.234078 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.237493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.234105 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.251424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.250617 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-dshm" pod "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" (UID: "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:05.251424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.250673 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-kube-api-access-qxx26" pod "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" (UID: "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507"). InnerVolumeSpecName "kube-api-access-qxx26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:05.251424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.250864 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tls-certs" pod "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" (UID: "caa69bc4-be2f-4acc-a8a2-4ba5ae42a507"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:05.254413 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.254388 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7"] Oct 30 06:22:05.254569 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.254515 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:05.335188 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.335149 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e07b5-328b-4d53-bb7d-c82b74d604d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-718730-c86996588-f9ss7\" (UID: \"788e07b5-328b-4d53-bb7d-c82b74d604d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:05.335406 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.335305 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grdq\" (UniqueName: \"kubernetes.io/projected/74387953-1299-4d3b-8e25-995bd29ce315-kube-api-access-6grdq\") pod \"payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm\" (UID: \"74387953-1299-4d3b-8e25-995bd29ce315\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:05.335493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.335399 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74387953-1299-4d3b-8e25-995bd29ce315-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm\" (UID: \"74387953-1299-4d3b-8e25-995bd29ce315\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:05.335604 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.335585 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmsw\" (UniqueName: \"kubernetes.io/projected/788e07b5-328b-4d53-bb7d-c82b74d604d6-kube-api-access-9pmsw\") pod \"payload-processing-e2e-ipp-b-718730-c86996588-f9ss7\" (UID: \"788e07b5-328b-4d53-bb7d-c82b74d604d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:05.335770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.335756 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.335846 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.335776 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxx26\" (UniqueName: \"kubernetes.io/projected/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-kube-api-access-qxx26\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.335846 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.335789 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.336297 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.336263 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74387953-1299-4d3b-8e25-995bd29ce315-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm\" (UID: \"74387953-1299-4d3b-8e25-995bd29ce315\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:05.357816 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.357786 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grdq\" (UniqueName: \"kubernetes.io/projected/74387953-1299-4d3b-8e25-995bd29ce315-kube-api-access-6grdq\") pod \"payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm\" (UID: \"74387953-1299-4d3b-8e25-995bd29ce315\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:05.437667 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.437615 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmsw\" (UniqueName: \"kubernetes.io/projected/788e07b5-328b-4d53-bb7d-c82b74d604d6-kube-api-access-9pmsw\") pod \"payload-processing-e2e-ipp-b-718730-c86996588-f9ss7\" (UID: \"788e07b5-328b-4d53-bb7d-c82b74d604d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:05.437863 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.437726 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e07b5-328b-4d53-bb7d-c82b74d604d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-718730-c86996588-f9ss7\" (UID: \"788e07b5-328b-4d53-bb7d-c82b74d604d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:05.438697 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.438673 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e07b5-328b-4d53-bb7d-c82b74d604d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-718730-c86996588-f9ss7\" (UID: \"788e07b5-328b-4d53-bb7d-c82b74d604d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:05.460238 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.460206 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmsw\" (UniqueName: \"kubernetes.io/projected/788e07b5-328b-4d53-bb7d-c82b74d604d6-kube-api-access-9pmsw\") pod \"payload-processing-e2e-ipp-b-718730-c86996588-f9ss7\" (UID: \"788e07b5-328b-4d53-bb7d-c82b74d604d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:05.569826 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.569787 5072 generic.go:356] "Generic (PLEG): container finished" podID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" containerID="894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb" exitCode=0 Oct 30 06:22:05.570027 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.569834 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" Oct 30 06:22:05.570027 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.569871 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" event={"ID":"9e70915b-1cdf-4270-81eb-5c7c8f9a5313","Type":"ContainerDied","Data":"894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb"} Oct 30 06:22:05.570027 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.569917 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj" event={"ID":"9e70915b-1cdf-4270-81eb-5c7c8f9a5313","Type":"ContainerDied","Data":"11462be43640f43d21ad30b85959356cc275321c5752e67e6f2d765ab5057a3d"} Oct 30 06:22:05.570027 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.569939 5072 scope.go:122] "RemoveContainer" containerID="894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb" Oct 30 06:22:05.571711 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.571689 5072 generic.go:356] "Generic (PLEG): container finished" podID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerID="00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490" exitCode=0 Oct 30 06:22:05.571828 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.571726 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" event={"ID":"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507","Type":"ContainerDied","Data":"00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490"} Oct 30 06:22:05.571828 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.571747 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" Oct 30 06:22:05.571828 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.571767 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" event={"ID":"caa69bc4-be2f-4acc-a8a2-4ba5ae42a507","Type":"ContainerDied","Data":"15b0ac7a7991ade895c5729a868c91c50de5cdb6a1e9b9e915d6e4f3f93fc146"} Oct 30 06:22:05.572532 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.572513 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:05.593042 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.581330 5072 scope.go:122] "RemoveContainer" containerID="894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb" Oct 30 06:22:05.593042 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:05.581810 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb\": container with ID starting with 894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb not found: ID does not exist" containerID="894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb" Oct 30 06:22:05.593042 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.581851 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb"} err="failed to get container status \"894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb\": rpc error: code = NotFound desc = could not find container \"894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb\": container with ID starting with 894f6506b6f91af33626e20c61c2779398911634cd25fbb7fb1e800ccc75e0bb not found: ID does not exist" Oct 30 06:22:05.593042 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.581875 5072 scope.go:122] "RemoveContainer" containerID="00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490" Oct 30 06:22:05.623726 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.623635 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:05.633706 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.633639 5072 scope.go:122] "RemoveContainer" containerID="00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490" Oct 30 06:22:05.634144 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:05.634069 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490\": container with ID starting with 00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490 not found: ID does not exist" containerID="00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490" Oct 30 06:22:05.634144 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.634105 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490"} err="failed to get container status \"00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490\": rpc error: code = NotFound desc = could not find container \"00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490\": container with ID starting with 00af847d4772fc2b0120799dbe03fdf82e4275c3c6bdfe35d3c80891b4549490 not found: ID does not exist" Oct 30 06:22:05.671011 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.668519 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj"] Oct 30 06:22:05.678522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.671830 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-2-w0-a14091-kserve-8f49f7666-vx9fj"] Oct 30 06:22:05.698312 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.698282 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b"] Oct 30 06:22:05.701959 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.701931 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b"] Oct 30 06:22:05.872502 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.872425 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/distinct-w0-a14091-kserve-56c886dd5c-hmn2b" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:22:05.897325 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.897302 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7"] Oct 30 06:22:05.898117 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:22:05.898091 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788e07b5_328b_4d53_bb7d_c82b74d604d6.slice/crio-2b110e89b443f7dc85c5f21e97e21c6368ad1aa2d2ace19ca652be65cf986366 WatchSource:0}: Error finding container 2b110e89b443f7dc85c5f21e97e21c6368ad1aa2d2ace19ca652be65cf986366: Status 404 returned error can't find the container with id 2b110e89b443f7dc85c5f21e97e21c6368ad1aa2d2ace19ca652be65cf986366 Oct 30 06:22:05.994209 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:05.994180 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm"] Oct 30 06:22:05.996250 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:22:05.996219 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74387953_1299_4d3b_8e25_995bd29ce315.slice/crio-761df4e8d1550130cb356dd9d1f71bb7182ebacd631ab8b60a75b7629a6ffc70 WatchSource:0}: Error finding container 761df4e8d1550130cb356dd9d1f71bb7182ebacd631ab8b60a75b7629a6ffc70: Status 404 returned error can't find the container with id 761df4e8d1550130cb356dd9d1f71bb7182ebacd631ab8b60a75b7629a6ffc70 Oct 30 06:22:06.520980 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.520946 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e70915b-1cdf-4270-81eb-5c7c8f9a5313" path="/var/lib/kubelet/pods/9e70915b-1cdf-4270-81eb-5c7c8f9a5313/volumes" Oct 30 06:22:06.521641 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.521621 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa69bc4-be2f-4acc-a8a2-4ba5ae42a507" path="/var/lib/kubelet/pods/caa69bc4-be2f-4acc-a8a2-4ba5ae42a507/volumes" Oct 30 06:22:06.588042 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.588000 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" event={"ID":"74387953-1299-4d3b-8e25-995bd29ce315","Type":"ContainerStarted","Data":"d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a"} Oct 30 06:22:06.588248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.588076 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:06.588248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.588100 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" event={"ID":"74387953-1299-4d3b-8e25-995bd29ce315","Type":"ContainerStarted","Data":"761df4e8d1550130cb356dd9d1f71bb7182ebacd631ab8b60a75b7629a6ffc70"} Oct 30 06:22:06.589909 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.589880 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" event={"ID":"788e07b5-328b-4d53-bb7d-c82b74d604d6","Type":"ContainerStarted","Data":"df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd"} Oct 30 06:22:06.590040 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.589929 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" event={"ID":"788e07b5-328b-4d53-bb7d-c82b74d604d6","Type":"ContainerStarted","Data":"2b110e89b443f7dc85c5f21e97e21c6368ad1aa2d2ace19ca652be65cf986366"} Oct 30 06:22:06.590040 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.589966 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:06.628576 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.628517 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" podStartSLOduration=2.628452181 podStartE2EDuration="2.628452181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:06.62606086 +0000 UTC m=+3086.229750122" watchObservedRunningTime="2026-10-30 06:22:06.628452181 +0000 UTC m=+3086.232141440" Oct 30 06:22:06.664753 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:06.664588 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" podStartSLOduration=1.66456917 podStartE2EDuration="1.66456917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:06.664013658 +0000 UTC m=+3086.267702916" watchObservedRunningTime="2026-10-30 06:22:06.66456917 +0000 UTC m=+3086.268258428" Oct 30 06:22:17.599210 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:17.599179 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:22:17.599696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:17.599254 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:22:19.303564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.303518 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm"] Oct 30 06:22:19.304190 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.303854 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" podUID="1bad623d-f3f5-4d28-be15-7512312eb5c6" containerName="maas-api" containerID="cri-o://690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f" gracePeriod=30 Oct 30 06:22:19.684481 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.682726 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:22:19.725494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.724551 5072 generic.go:356] "Generic (PLEG): container finished" podID="1bad623d-f3f5-4d28-be15-7512312eb5c6" containerID="690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f" exitCode=0 Oct 30 06:22:19.725494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.724738 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" event={"ID":"1bad623d-f3f5-4d28-be15-7512312eb5c6","Type":"ContainerDied","Data":"690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f"} Oct 30 06:22:19.725494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.724788 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" event={"ID":"1bad623d-f3f5-4d28-be15-7512312eb5c6","Type":"ContainerDied","Data":"994f649a7891283592dd9a983016cd70eb420966b16ec4bac9bb160574cfd664"} Oct 30 06:22:19.725494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.724817 5072 scope.go:122] "RemoveContainer" containerID="690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f" Oct 30 06:22:19.725494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.724989 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm" Oct 30 06:22:19.770371 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.769929 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls\") pod \"1bad623d-f3f5-4d28-be15-7512312eb5c6\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " Oct 30 06:22:19.771041 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.770660 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-metrics-certs\") pod \"1bad623d-f3f5-4d28-be15-7512312eb5c6\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " Oct 30 06:22:19.801836 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.771218 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bad623d-f3f5-4d28-be15-7512312eb5c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bad623d-f3f5-4d28-be15-7512312eb5c6-openshift-service-ca\") pod \"1bad623d-f3f5-4d28-be15-7512312eb5c6\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " Oct 30 06:22:19.801836 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.801770 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad623d-f3f5-4d28-be15-7512312eb5c6-openshift-service-ca" pod "1bad623d-f3f5-4d28-be15-7512312eb5c6" (UID: "1bad623d-f3f5-4d28-be15-7512312eb5c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:19.824897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.802586 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bad623d-f3f5-4d28-be15-7512312eb5c6-kube-api-access-kmjjz\" (UniqueName: \"kubernetes.io/projected/1bad623d-f3f5-4d28-be15-7512312eb5c6-kube-api-access-kmjjz\") pod \"1bad623d-f3f5-4d28-be15-7512312eb5c6\" (UID: \"1bad623d-f3f5-4d28-be15-7512312eb5c6\") " Oct 30 06:22:19.824897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.803398 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bad623d-f3f5-4d28-be15-7512312eb5c6-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:19.824897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.804642 5072 scope.go:122] "RemoveContainer" containerID="690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f" Oct 30 06:22:19.824897 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:19.805174 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f\": container with ID starting with 690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f not found: ID does not exist" containerID="690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f" Oct 30 06:22:19.824897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.805234 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f"} err="failed to get container status \"690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f\": rpc error: code = NotFound desc = could not find container \"690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f\": container with ID starting with 690d246d84b8f6254b7204c4ef4d567e522c9be27e995d1020a3b391eb209c1f not found: ID does not exist" Oct 30 06:22:19.848156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.832616 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls" pod "1bad623d-f3f5-4d28-be15-7512312eb5c6" (UID: "1bad623d-f3f5-4d28-be15-7512312eb5c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:19.848156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.833948 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-metrics-certs" pod "1bad623d-f3f5-4d28-be15-7512312eb5c6" (UID: "1bad623d-f3f5-4d28-be15-7512312eb5c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:19.848156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.834203 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bad623d-f3f5-4d28-be15-7512312eb5c6-kube-api-access-kmjjz" pod "1bad623d-f3f5-4d28-be15-7512312eb5c6" (UID: "1bad623d-f3f5-4d28-be15-7512312eb5c6"). InnerVolumeSpecName "kube-api-access-kmjjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:19.905640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.905556 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:19.905640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.905592 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bad623d-f3f5-4d28-be15-7512312eb5c6-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:19.905640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:19.905608 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmjjz\" (UniqueName: \"kubernetes.io/projected/1bad623d-f3f5-4d28-be15-7512312eb5c6-kube-api-access-kmjjz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.071814 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:20.071783 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm"] Oct 30 06:22:20.076835 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:20.076811 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-080391-57c48f5545-fxdgm"] Oct 30 06:22:20.521984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:20.521951 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bad623d-f3f5-4d28-be15-7512312eb5c6" path="/var/lib/kubelet/pods/1bad623d-f3f5-4d28-be15-7512312eb5c6/volumes" Oct 30 06:22:21.930248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:21.930209 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn"] Oct 30 06:22:21.930788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:21.930480 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" podUID="84881af6-829e-4489-9c91-319b111ef962" containerName="maas-api" containerID="cri-o://da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc" gracePeriod=30 Oct 30 06:22:22.129740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.129702 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8"] Oct 30 06:22:22.130367 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.130334 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" podUID="52ba9fdd-9ed9-488c-ae0b-5d56573bfd46" containerName="payload-pre-processing" containerID="cri-o://e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a" gracePeriod=30 Oct 30 06:22:22.296871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.296832 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:22:22.354946 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.354903 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-metrics-certs\") pod \"84881af6-829e-4489-9c91-319b111ef962\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " Oct 30 06:22:22.355146 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.354981 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-maas-api-tls\") pod \"84881af6-829e-4489-9c91-319b111ef962\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " Oct 30 06:22:22.355146 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.355091 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84881af6-829e-4489-9c91-319b111ef962-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84881af6-829e-4489-9c91-319b111ef962-openshift-service-ca\") pod \"84881af6-829e-4489-9c91-319b111ef962\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " Oct 30 06:22:22.355263 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.355181 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84881af6-829e-4489-9c91-319b111ef962-kube-api-access-gqvlz\" (UniqueName: \"kubernetes.io/projected/84881af6-829e-4489-9c91-319b111ef962-kube-api-access-gqvlz\") pod \"84881af6-829e-4489-9c91-319b111ef962\" (UID: \"84881af6-829e-4489-9c91-319b111ef962\") " Oct 30 06:22:22.359704 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.359663 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84881af6-829e-4489-9c91-319b111ef962-openshift-service-ca" pod "84881af6-829e-4489-9c91-319b111ef962" (UID: "84881af6-829e-4489-9c91-319b111ef962"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:22.364085 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.364048 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-metrics-certs" pod "84881af6-829e-4489-9c91-319b111ef962" (UID: "84881af6-829e-4489-9c91-319b111ef962"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:22.364668 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.364638 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-maas-api-tls" pod "84881af6-829e-4489-9c91-319b111ef962" (UID: "84881af6-829e-4489-9c91-319b111ef962"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:22.369603 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.369573 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84881af6-829e-4489-9c91-319b111ef962-kube-api-access-gqvlz" pod "84881af6-829e-4489-9c91-319b111ef962" (UID: "84881af6-829e-4489-9c91-319b111ef962"). InnerVolumeSpecName "kube-api-access-gqvlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:22.419512 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.418506 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:22:22.499373 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.495984 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.499373 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.496016 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84881af6-829e-4489-9c91-319b111ef962-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.499373 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.496031 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84881af6-829e-4489-9c91-319b111ef962-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.499373 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.496047 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqvlz\" (UniqueName: \"kubernetes.io/projected/84881af6-829e-4489-9c91-319b111ef962-kube-api-access-gqvlz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.598561 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.598441 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-plugins-config-volume\") pod \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\" (UID: \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\") " Oct 30 06:22:22.600167 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.600132 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-kube-api-access-dz7n8\" (UniqueName: \"kubernetes.io/projected/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-kube-api-access-dz7n8\") pod \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\" (UID: \"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46\") " Oct 30 06:22:22.601746 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.600429 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-plugins-config-volume" pod "52ba9fdd-9ed9-488c-ae0b-5d56573bfd46" (UID: "52ba9fdd-9ed9-488c-ae0b-5d56573bfd46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:22.656048 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.656006 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-kube-api-access-dz7n8" pod "52ba9fdd-9ed9-488c-ae0b-5d56573bfd46" (UID: "52ba9fdd-9ed9-488c-ae0b-5d56573bfd46"). InnerVolumeSpecName "kube-api-access-dz7n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:22.702209 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.702174 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz7n8\" (UniqueName: \"kubernetes.io/projected/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-kube-api-access-dz7n8\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.702209 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.702204 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.754827 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.754790 5072 generic.go:356] "Generic (PLEG): container finished" podID="84881af6-829e-4489-9c91-319b111ef962" containerID="da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc" exitCode=0 Oct 30 06:22:22.755062 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.754863 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" Oct 30 06:22:22.755062 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.754890 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" event={"ID":"84881af6-829e-4489-9c91-319b111ef962","Type":"ContainerDied","Data":"da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc"} Oct 30 06:22:22.755062 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.754956 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn" event={"ID":"84881af6-829e-4489-9c91-319b111ef962","Type":"ContainerDied","Data":"89001d7dfc4eb649863463b775c26f46bf9129eba2c1a229770d1a6a335d0ee4"} Oct 30 06:22:22.755062 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.754996 5072 scope.go:122] "RemoveContainer" containerID="da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc" Oct 30 06:22:22.757414 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.757347 5072 generic.go:356] "Generic (PLEG): container finished" podID="52ba9fdd-9ed9-488c-ae0b-5d56573bfd46" containerID="e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a" exitCode=0 Oct 30 06:22:22.757581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.757566 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" event={"ID":"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46","Type":"ContainerDied","Data":"e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a"} Oct 30 06:22:22.757658 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.757611 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" event={"ID":"52ba9fdd-9ed9-488c-ae0b-5d56573bfd46","Type":"ContainerDied","Data":"c919801cf527a5a97410e5ea5fdda3456b5c3d62f2e67d48d84e74bfe3c8f9c1"} Oct 30 06:22:22.757720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.757677 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8" Oct 30 06:22:22.806676 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.806648 5072 scope.go:122] "RemoveContainer" containerID="da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc" Oct 30 06:22:22.807561 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:22.807338 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc\": container with ID starting with da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc not found: ID does not exist" containerID="da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc" Oct 30 06:22:22.807561 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.807383 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc"} err="failed to get container status \"da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc\": rpc error: code = NotFound desc = could not find container \"da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc\": container with ID starting with da68e549f7caf9590e2c556d6c663a2cc5ea4a8b2055c8147c2e8c6683786bdc not found: ID does not exist" Oct 30 06:22:22.807561 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.807409 5072 scope.go:122] "RemoveContainer" containerID="e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a" Oct 30 06:22:22.825715 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.825687 5072 scope.go:122] "RemoveContainer" containerID="e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a" Oct 30 06:22:22.826261 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:22.826211 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a\": container with ID starting with e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a not found: ID does not exist" containerID="e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a" Oct 30 06:22:22.826444 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.826413 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a"} err="failed to get container status \"e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a\": rpc error: code = NotFound desc = could not find container \"e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a\": container with ID starting with e547a3f9280a7be7371e4eabf33a4e9d3a78a3a97553a3d758ff29581b71dd5a not found: ID does not exist" Oct 30 06:22:22.832886 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.832862 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn"] Oct 30 06:22:22.841024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.840999 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07b278-569b76f8d7-44lhn"] Oct 30 06:22:22.871262 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.871234 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8"] Oct 30 06:22:22.876140 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:22.876113 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-75bf8cb995-pxbm8"] Oct 30 06:22:24.014175 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.014137 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq"] Oct 30 06:22:24.014870 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.014852 5072 cpu_manager.go:414] "containerMap: removing container" podUID="52ba9fdd-9ed9-488c-ae0b-5d56573bfd46" containerName="payload-pre-processing" Oct 30 06:22:24.014930 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.014872 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ba9fdd-9ed9-488c-ae0b-5d56573bfd46" containerName="payload-pre-processing" Oct 30 06:22:24.014930 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.014907 5072 cpu_manager.go:414] "containerMap: removing container" podUID="84881af6-829e-4489-9c91-319b111ef962" containerName="maas-api" Oct 30 06:22:24.014930 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.014913 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84881af6-829e-4489-9c91-319b111ef962" containerName="maas-api" Oct 30 06:22:24.014930 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.014922 5072 cpu_manager.go:414] "containerMap: removing container" podUID="1bad623d-f3f5-4d28-be15-7512312eb5c6" containerName="maas-api" Oct 30 06:22:24.014930 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.014927 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bad623d-f3f5-4d28-be15-7512312eb5c6" containerName="maas-api" Oct 30 06:22:24.015114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.015032 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="84881af6-829e-4489-9c91-319b111ef962" containerName="maas-api" Oct 30 06:22:24.015114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.015040 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bad623d-f3f5-4d28-be15-7512312eb5c6" containerName="maas-api" Oct 30 06:22:24.015114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.015051 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ba9fdd-9ed9-488c-ae0b-5d56573bfd46" containerName="payload-pre-processing" Oct 30 06:22:24.019897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.019877 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.053313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.053280 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq"] Oct 30 06:22:24.085498 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.085427 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a14091-openshift-default-dockercfg-xdwgh\"" Oct 30 06:22:24.126255 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.126218 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.126255 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.126267 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.126567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.126294 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.126567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.126356 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pg9x\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-kube-api-access-8pg9x\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.126695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.126587 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.126695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.126686 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54f773d-5f0c-4bde-a25f-5385934a95e1-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.126797 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.126754 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.126797 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.126773 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.126922 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.126892 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228158 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228117 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228158 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228169 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228441 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228187 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228441 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228208 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pg9x\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-kube-api-access-8pg9x\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228441 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228338 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228441 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228402 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54f773d-5f0c-4bde-a25f-5385934a95e1-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228709 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228448 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228709 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228489 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228709 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228566 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.228890 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.228868 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.229066 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.229040 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.229273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.229107 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.229273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.229191 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.229401 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.229347 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54f773d-5f0c-4bde-a25f-5385934a95e1-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.230860 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.230837 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.231060 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.231041 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.250615 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.250591 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.250735 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.250721 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pg9x\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-kube-api-access-8pg9x\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.336204 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.336107 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:24.571261 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.569741 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ba9fdd-9ed9-488c-ae0b-5d56573bfd46" path="/var/lib/kubelet/pods/52ba9fdd-9ed9-488c-ae0b-5d56573bfd46/volumes" Oct 30 06:22:24.571261 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.570386 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84881af6-829e-4489-9c91-319b111ef962" path="/var/lib/kubelet/pods/84881af6-829e-4489-9c91-319b111ef962/volumes" Oct 30 06:22:24.760753 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.760695 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq"] Oct 30 06:22:24.796413 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:22:24.796380 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54f773d_5f0c_4bde_a25f_5385934a95e1.slice/crio-41fdc5d7d38a8f35fd7ae1958d32cf937f58dc8821069f0f65be91594db33ed5 WatchSource:0}: Error finding container 41fdc5d7d38a8f35fd7ae1958d32cf937f58dc8821069f0f65be91594db33ed5: Status 404 returned error can't find the container with id 41fdc5d7d38a8f35fd7ae1958d32cf937f58dc8821069f0f65be91594db33ed5 Oct 30 06:22:24.799085 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.799053 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:22:24.799163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.799122 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:22:24.799163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:24.799151 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:22:25.849673 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:25.849632 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" event={"ID":"c54f773d-5f0c-4bde-a25f-5385934a95e1","Type":"ContainerStarted","Data":"1f1312f304f7a75d3799a6e3b9a07d23bd7d3a10e821740fc5e13395d5c504b3"} Oct 30 06:22:25.850079 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:25.849718 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" event={"ID":"c54f773d-5f0c-4bde-a25f-5385934a95e1","Type":"ContainerStarted","Data":"41fdc5d7d38a8f35fd7ae1958d32cf937f58dc8821069f0f65be91594db33ed5"} Oct 30 06:22:25.929484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:25.925649 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" podStartSLOduration=2.92562833 podStartE2EDuration="2.92562833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:25.902435157 +0000 UTC m=+3105.506124431" watchObservedRunningTime="2026-10-30 06:22:25.92562833 +0000 UTC m=+3105.529317588" Oct 30 06:22:26.338742 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:26.338614 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:26.339083 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:26.339056 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:22:26.339296 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:26.339266 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 06:22:27.337384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:27.337346 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:22:27.337869 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:27.337403 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 06:22:28.263164 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:28.263119 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2"] Oct 30 06:22:28.263923 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:28.263712 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" podUID="12551d54-2727-4931-8ac4-4023d7613f83" containerName="istio-proxy" containerID="cri-o://6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e" gracePeriod=30 Oct 30 06:22:28.343200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:28.343164 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:22:28.344638 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:28.343845 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 06:22:29.337488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:29.336865 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:22:29.337488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:29.336933 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 06:22:30.379863 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:30.379824 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:22:30.380533 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:30.379887 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 06:22:30.740539 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:30.732645 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:22:31.336997 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:31.336958 5072 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:22:31.337231 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:31.337038 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 06:22:31.753883 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:31.753834 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc"] Oct 30 06:22:31.754384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:31.754086 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" podUID="76a5be12-b842-446d-b841-3af5bf5dde44" containerName="maas-api" containerID="cri-o://5e8e7e3e2aa631876bc484aa80f96416982f65f5f97d78d56680fb9eb29a0824" gracePeriod=30 Oct 30 06:22:31.901084 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:31.901035 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl"] Oct 30 06:22:31.901281 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:31.901259 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" podUID="2b0d06fe-f98e-4c1b-8669-aa8ee23164f5" containerName="payload-processing" containerID="cri-o://08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4" gracePeriod=30 Oct 30 06:22:31.914694 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:31.914660 5072 generic.go:356] "Generic (PLEG): container finished" podID="76a5be12-b842-446d-b841-3af5bf5dde44" containerID="5e8e7e3e2aa631876bc484aa80f96416982f65f5f97d78d56680fb9eb29a0824" exitCode=0 Oct 30 06:22:31.914837 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:31.914733 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" event={"ID":"76a5be12-b842-446d-b841-3af5bf5dde44","Type":"ContainerDied","Data":"5e8e7e3e2aa631876bc484aa80f96416982f65f5f97d78d56680fb9eb29a0824"} Oct 30 06:22:32.296530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.287780 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:22:32.396724 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.396670 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:22:32.421508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.419008 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-metrics-certs\") pod \"76a5be12-b842-446d-b841-3af5bf5dde44\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " Oct 30 06:22:32.421508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.419173 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a5be12-b842-446d-b841-3af5bf5dde44-kube-api-access-zj7w6\" (UniqueName: \"kubernetes.io/projected/76a5be12-b842-446d-b841-3af5bf5dde44-kube-api-access-zj7w6\") pod \"76a5be12-b842-446d-b841-3af5bf5dde44\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " Oct 30 06:22:32.421508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.421181 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a5be12-b842-446d-b841-3af5bf5dde44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a5be12-b842-446d-b841-3af5bf5dde44-openshift-service-ca\") pod \"76a5be12-b842-446d-b841-3af5bf5dde44\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " Oct 30 06:22:32.421832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.421626 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a5be12-b842-446d-b841-3af5bf5dde44-openshift-service-ca" pod "76a5be12-b842-446d-b841-3af5bf5dde44" (UID: "76a5be12-b842-446d-b841-3af5bf5dde44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:32.421832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.421746 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-maas-api-tls\") pod \"76a5be12-b842-446d-b841-3af5bf5dde44\" (UID: \"76a5be12-b842-446d-b841-3af5bf5dde44\") " Oct 30 06:22:32.423790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.422780 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76a5be12-b842-446d-b841-3af5bf5dde44-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:32.423790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.423453 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a5be12-b842-446d-b841-3af5bf5dde44-kube-api-access-zj7w6" pod "76a5be12-b842-446d-b841-3af5bf5dde44" (UID: "76a5be12-b842-446d-b841-3af5bf5dde44"). InnerVolumeSpecName "kube-api-access-zj7w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:32.426647 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.426613 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-metrics-certs" pod "76a5be12-b842-446d-b841-3af5bf5dde44" (UID: "76a5be12-b842-446d-b841-3af5bf5dde44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:32.430976 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.430938 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-maas-api-tls" pod "76a5be12-b842-446d-b841-3af5bf5dde44" (UID: "76a5be12-b842-446d-b841-3af5bf5dde44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:32.525306 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.525260 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-kube-api-access-hqrqc\" (UniqueName: \"kubernetes.io/projected/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-kube-api-access-hqrqc\") pod \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\" (UID: \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\") " Oct 30 06:22:32.525606 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.525570 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-plugins-config-volume\") pod \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\" (UID: \"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5\") " Oct 30 06:22:32.525958 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.525929 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-plugins-config-volume" pod "2b0d06fe-f98e-4c1b-8669-aa8ee23164f5" (UID: "2b0d06fe-f98e-4c1b-8669-aa8ee23164f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:32.526107 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.526083 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:32.526107 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.526103 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:32.526244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.526118 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj7w6\" (UniqueName: \"kubernetes.io/projected/76a5be12-b842-446d-b841-3af5bf5dde44-kube-api-access-zj7w6\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:32.526244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.526131 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76a5be12-b842-446d-b841-3af5bf5dde44-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:32.528101 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.528076 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-kube-api-access-hqrqc" pod "2b0d06fe-f98e-4c1b-8669-aa8ee23164f5" (UID: "2b0d06fe-f98e-4c1b-8669-aa8ee23164f5"). InnerVolumeSpecName "kube-api-access-hqrqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:32.627946 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.627915 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqrqc\" (UniqueName: \"kubernetes.io/projected/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5-kube-api-access-hqrqc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:32.760110 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.759976 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:32.760977 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.760800 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:32.762511 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.762442 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:22:32.926688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.926599 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" Oct 30 06:22:32.926860 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.926598 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc" event={"ID":"76a5be12-b842-446d-b841-3af5bf5dde44","Type":"ContainerDied","Data":"2586d433e18eea7e15a55fdda3d0090273fd243cda25abe1aa662a1e9415690b"} Oct 30 06:22:32.926860 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.926821 5072 scope.go:122] "RemoveContainer" containerID="5e8e7e3e2aa631876bc484aa80f96416982f65f5f97d78d56680fb9eb29a0824" Oct 30 06:22:32.930075 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.930036 5072 generic.go:356] "Generic (PLEG): container finished" podID="2b0d06fe-f98e-4c1b-8669-aa8ee23164f5" containerID="08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4" exitCode=0 Oct 30 06:22:32.930191 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.930102 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" Oct 30 06:22:32.930191 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.930159 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" event={"ID":"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5","Type":"ContainerDied","Data":"08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4"} Oct 30 06:22:32.930268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.930202 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl" event={"ID":"2b0d06fe-f98e-4c1b-8669-aa8ee23164f5","Type":"ContainerDied","Data":"844864be4076d17d4e36796c10fbf86291379dbf793fb6706cd1c3ebe42069ac"} Oct 30 06:22:32.943030 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.943005 5072 scope.go:122] "RemoveContainer" containerID="08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4" Oct 30 06:22:32.968523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.968498 5072 scope.go:122] "RemoveContainer" containerID="08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4" Oct 30 06:22:32.968967 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:32.968938 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4\": container with ID starting with 08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4 not found: ID does not exist" containerID="08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4" Oct 30 06:22:32.969045 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.968981 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4"} err="failed to get container status \"08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4\": rpc error: code = NotFound desc = could not find container \"08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4\": container with ID starting with 08ff48f788234e0c4abe785bd5c21d6275942b84cf7dd454a5c4293a0d8155c4 not found: ID does not exist" Oct 30 06:22:32.985297 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.985256 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc"] Oct 30 06:22:32.991520 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:32.991489 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8b892-6f9ccf89fd-7pcqc"] Oct 30 06:22:33.011833 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.011778 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl"] Oct 30 06:22:33.015520 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.015495 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-5544768d96-89rgl"] Oct 30 06:22:33.586055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.586022 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:22:33.743757 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.743719 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-data\") pod \"12551d54-2727-4931-8ac4-4023d7613f83\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " Oct 30 06:22:33.743937 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.743825 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-certs\") pod \"12551d54-2727-4931-8ac4-4023d7613f83\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " Oct 30 06:22:33.743937 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.743907 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12551d54-2727-4931-8ac4-4023d7613f83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12551d54-2727-4931-8ac4-4023d7613f83-istio-podinfo\") pod \"12551d54-2727-4931-8ac4-4023d7613f83\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " Oct 30 06:22:33.744086 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744005 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12551d54-2727-4931-8ac4-4023d7613f83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12551d54-2727-4931-8ac4-4023d7613f83-istiod-ca-cert\") pod \"12551d54-2727-4931-8ac4-4023d7613f83\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " Oct 30 06:22:33.744086 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744044 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-socket\") pod \"12551d54-2727-4931-8ac4-4023d7613f83\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " Oct 30 06:22:33.744207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744083 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-data" pod "12551d54-2727-4931-8ac4-4023d7613f83" (UID: "12551d54-2727-4931-8ac4-4023d7613f83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.744207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744179 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-kube-api-access-rf5xt\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-kube-api-access-rf5xt\") pod \"12551d54-2727-4931-8ac4-4023d7613f83\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " Oct 30 06:22:33.744207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744194 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-certs" pod "12551d54-2727-4931-8ac4-4023d7613f83" (UID: "12551d54-2727-4931-8ac4-4023d7613f83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.744332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744248 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-envoy\") pod \"12551d54-2727-4931-8ac4-4023d7613f83\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " Oct 30 06:22:33.744332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744296 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-credential-socket\") pod \"12551d54-2727-4931-8ac4-4023d7613f83\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " Oct 30 06:22:33.744441 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744357 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-istio-token\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-istio-token\") pod \"12551d54-2727-4931-8ac4-4023d7613f83\" (UID: \"12551d54-2727-4931-8ac4-4023d7613f83\") " Oct 30 06:22:33.744803 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744775 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-credential-socket" pod "12551d54-2727-4931-8ac4-4023d7613f83" (UID: "12551d54-2727-4931-8ac4-4023d7613f83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.744899 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744819 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12551d54-2727-4931-8ac4-4023d7613f83-istiod-ca-cert" pod "12551d54-2727-4931-8ac4-4023d7613f83" (UID: "12551d54-2727-4931-8ac4-4023d7613f83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:33.744991 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.744970 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-socket" pod "12551d54-2727-4931-8ac4-4023d7613f83" (UID: "12551d54-2727-4931-8ac4-4023d7613f83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.745436 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.745414 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.745598 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.745582 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12551d54-2727-4931-8ac4-4023d7613f83-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.745699 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.745686 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.745792 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.745779 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.745937 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.745923 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.747494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.747425 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-envoy" pod "12551d54-2727-4931-8ac4-4023d7613f83" (UID: "12551d54-2727-4931-8ac4-4023d7613f83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.747591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.747504 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12551d54-2727-4931-8ac4-4023d7613f83-istio-podinfo" pod "12551d54-2727-4931-8ac4-4023d7613f83" (UID: "12551d54-2727-4931-8ac4-4023d7613f83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:33.747761 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.747739 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-kube-api-access-rf5xt" pod "12551d54-2727-4931-8ac4-4023d7613f83" (UID: "12551d54-2727-4931-8ac4-4023d7613f83"). InnerVolumeSpecName "kube-api-access-rf5xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:33.747761 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.747753 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-istio-token" pod "12551d54-2727-4931-8ac4-4023d7613f83" (UID: "12551d54-2727-4931-8ac4-4023d7613f83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:33.847657 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.847562 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12551d54-2727-4931-8ac4-4023d7613f83-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.847657 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.847594 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5xt\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-kube-api-access-rf5xt\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.847657 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.847606 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12551d54-2727-4931-8ac4-4023d7613f83-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.847657 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.847615 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12551d54-2727-4931-8ac4-4023d7613f83-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.940815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.940774 5072 generic.go:356] "Generic (PLEG): container finished" podID="12551d54-2727-4931-8ac4-4023d7613f83" containerID="6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e" exitCode=0 Oct 30 06:22:33.941016 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.940825 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" event={"ID":"12551d54-2727-4931-8ac4-4023d7613f83","Type":"ContainerDied","Data":"6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e"} Oct 30 06:22:33.941016 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.940859 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" Oct 30 06:22:33.941016 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.940903 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2" event={"ID":"12551d54-2727-4931-8ac4-4023d7613f83","Type":"ContainerDied","Data":"7d8de0546db51aea5eb4223eb7f015d6a84b8f18456dc5fc1181c33c2e0ced38"} Oct 30 06:22:33.941016 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.940934 5072 scope.go:122] "RemoveContainer" containerID="6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e" Oct 30 06:22:33.955166 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.955143 5072 scope.go:122] "RemoveContainer" containerID="6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e" Oct 30 06:22:33.955510 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:33.955446 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e\": container with ID starting with 6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e not found: ID does not exist" containerID="6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e" Oct 30 06:22:33.955616 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:33.955520 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e"} err="failed to get container status \"6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e\": rpc error: code = NotFound desc = could not find container \"6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e\": container with ID starting with 6ce8e61f7bceda3a751974e877c9ca8f668169829d5b5d924e8d4082ee85ec6e not found: ID does not exist" Oct 30 06:22:34.035826 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:34.035793 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2"] Oct 30 06:22:34.044934 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:34.044898 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-sq8n2"] Oct 30 06:22:34.536358 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:34.536320 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12551d54-2727-4931-8ac4-4023d7613f83" path="/var/lib/kubelet/pods/12551d54-2727-4931-8ac4-4023d7613f83/volumes" Oct 30 06:22:34.536897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:34.536878 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0d06fe-f98e-4c1b-8669-aa8ee23164f5" path="/var/lib/kubelet/pods/2b0d06fe-f98e-4c1b-8669-aa8ee23164f5/volumes" Oct 30 06:22:34.537358 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:34.537343 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a5be12-b842-446d-b841-3af5bf5dde44" path="/var/lib/kubelet/pods/76a5be12-b842-446d-b841-3af5bf5dde44/volumes" Oct 30 06:22:39.527483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:39.519433 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2"] Oct 30 06:22:39.527483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:39.519704 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" containerID="cri-o://c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f" gracePeriod=30 Oct 30 06:22:44.825725 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.825690 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.928613 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istiod-ca-cert\") pod \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.928686 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-podinfo\") pod \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.928856 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-token\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-token\") pod \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.928918 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-envoy\") pod \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.929010 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-data\") pod \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.929059 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-kube-api-access-bflvp\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-kube-api-access-bflvp\") pod \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.929129 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-credential-socket\") pod \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.929177 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-certs\") pod \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.929308 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-socket\") pod \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\" (UID: \"29d97d10-ff5d-499a-a994-2721e2e3c8fa\") " Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.930195 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-socket" pod "29d97d10-ff5d-499a-a994-2721e2e3c8fa" (UID: "29d97d10-ff5d-499a-a994-2721e2e3c8fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.930563 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istiod-ca-cert" pod "29d97d10-ff5d-499a-a994-2721e2e3c8fa" (UID: "29d97d10-ff5d-499a-a994-2721e2e3c8fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.931292 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-data" pod "29d97d10-ff5d-499a-a994-2721e2e3c8fa" (UID: "29d97d10-ff5d-499a-a994-2721e2e3c8fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.931592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.931550 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-credential-socket" pod "29d97d10-ff5d-499a-a994-2721e2e3c8fa" (UID: "29d97d10-ff5d-499a-a994-2721e2e3c8fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.932492 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.931720 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-certs" pod "29d97d10-ff5d-499a-a994-2721e2e3c8fa" (UID: "29d97d10-ff5d-499a-a994-2721e2e3c8fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.962802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.962211 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-podinfo" pod "29d97d10-ff5d-499a-a994-2721e2e3c8fa" (UID: "29d97d10-ff5d-499a-a994-2721e2e3c8fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:44.962802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.962428 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-token" pod "29d97d10-ff5d-499a-a994-2721e2e3c8fa" (UID: "29d97d10-ff5d-499a-a994-2721e2e3c8fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:44.962802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.962765 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-envoy" pod "29d97d10-ff5d-499a-a994-2721e2e3c8fa" (UID: "29d97d10-ff5d-499a-a994-2721e2e3c8fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.963080 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:44.962983 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-kube-api-access-bflvp" pod "29d97d10-ff5d-499a-a994-2721e2e3c8fa" (UID: "29d97d10-ff5d-499a-a994-2721e2e3c8fa"). InnerVolumeSpecName "kube-api-access-bflvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:45.030877 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.030847 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.030877 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.030874 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.030877 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.030883 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.031259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.030893 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.031259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.030900 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.031259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.030908 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bflvp\" (UniqueName: \"kubernetes.io/projected/29d97d10-ff5d-499a-a994-2721e2e3c8fa-kube-api-access-bflvp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.031259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.030917 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.031259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.030925 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.031259 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.030933 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d97d10-ff5d-499a-a994-2721e2e3c8fa-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.042860 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.042781 5072 generic.go:356] "Generic (PLEG): container finished" podID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerID="c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f" exitCode=0 Oct 30 06:22:45.042860 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.042842 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" Oct 30 06:22:45.043042 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.042865 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" event={"ID":"29d97d10-ff5d-499a-a994-2721e2e3c8fa","Type":"ContainerDied","Data":"c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f"} Oct 30 06:22:45.043042 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.042933 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2" event={"ID":"29d97d10-ff5d-499a-a994-2721e2e3c8fa","Type":"ContainerDied","Data":"e2ce6184eb2400c9932fb4f24d6bebb28e725bc7d2d702260903c4a0341efc3b"} Oct 30 06:22:45.043042 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.042958 5072 scope.go:122] "RemoveContainer" containerID="c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f" Oct 30 06:22:45.055620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.055594 5072 scope.go:122] "RemoveContainer" containerID="97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e" Oct 30 06:22:45.068776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.068753 5072 scope.go:122] "RemoveContainer" containerID="c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f" Oct 30 06:22:45.069083 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:45.069063 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f\": container with ID starting with c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f not found: ID does not exist" containerID="c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f" Oct 30 06:22:45.069143 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.069092 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f"} err="failed to get container status \"c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f\": rpc error: code = NotFound desc = could not find container \"c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f\": container with ID starting with c433174876ab81480ae4eee23feddfd715a840bacaa3cb0d51cd3d2728eccd2f not found: ID does not exist" Oct 30 06:22:45.069143 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.069109 5072 scope.go:122] "RemoveContainer" containerID="97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e" Oct 30 06:22:45.069419 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:45.069393 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e\": container with ID starting with 97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e not found: ID does not exist" containerID="97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e" Oct 30 06:22:45.069498 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.069429 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e"} err="failed to get container status \"97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e\": rpc error: code = NotFound desc = could not find container \"97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e\": container with ID starting with 97d8cb61bc10a7b69e956a2581bb01ffe92eccb459c7e0943c0ab21a6f07df8e not found: ID does not exist" Oct 30 06:22:45.096494 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.096453 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2"] Oct 30 06:22:45.100070 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:45.100047 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8b892-openshift-default-5b8466f85-8n7m2"] Oct 30 06:22:46.520100 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:46.520064 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" path="/var/lib/kubelet/pods/29d97d10-ff5d-499a-a994-2721e2e3c8fa/volumes" Oct 30 06:22:47.088497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.086924 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl"] Oct 30 06:22:47.088497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.087186 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" podUID="9a235824-ceb8-4424-9404-64c98bd44b39" containerName="maas-api" containerID="cri-o://d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4" gracePeriod=30 Oct 30 06:22:47.432719 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.432689 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:22:47.565117 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.565080 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-maas-api-tls\") pod \"9a235824-ceb8-4424-9404-64c98bd44b39\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " Oct 30 06:22:47.565695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.565166 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-metrics-certs\") pod \"9a235824-ceb8-4424-9404-64c98bd44b39\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " Oct 30 06:22:47.565695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.565216 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a235824-ceb8-4424-9404-64c98bd44b39-kube-api-access-gsgmx\" (UniqueName: \"kubernetes.io/projected/9a235824-ceb8-4424-9404-64c98bd44b39-kube-api-access-gsgmx\") pod \"9a235824-ceb8-4424-9404-64c98bd44b39\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " Oct 30 06:22:47.565695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.565330 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a235824-ceb8-4424-9404-64c98bd44b39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a235824-ceb8-4424-9404-64c98bd44b39-openshift-service-ca\") pod \"9a235824-ceb8-4424-9404-64c98bd44b39\" (UID: \"9a235824-ceb8-4424-9404-64c98bd44b39\") " Oct 30 06:22:47.566065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.566030 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a235824-ceb8-4424-9404-64c98bd44b39-openshift-service-ca" pod "9a235824-ceb8-4424-9404-64c98bd44b39" (UID: "9a235824-ceb8-4424-9404-64c98bd44b39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:47.570620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.570589 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-metrics-certs" pod "9a235824-ceb8-4424-9404-64c98bd44b39" (UID: "9a235824-ceb8-4424-9404-64c98bd44b39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:47.570737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.570636 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a235824-ceb8-4424-9404-64c98bd44b39-kube-api-access-gsgmx" pod "9a235824-ceb8-4424-9404-64c98bd44b39" (UID: "9a235824-ceb8-4424-9404-64c98bd44b39"). InnerVolumeSpecName "kube-api-access-gsgmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:47.570898 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.570868 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-maas-api-tls" pod "9a235824-ceb8-4424-9404-64c98bd44b39" (UID: "9a235824-ceb8-4424-9404-64c98bd44b39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:47.667102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.667014 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a235824-ceb8-4424-9404-64c98bd44b39-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.667102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.667052 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.667102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.667061 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a235824-ceb8-4424-9404-64c98bd44b39-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.667102 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:47.667069 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsgmx\" (UniqueName: \"kubernetes.io/projected/9a235824-ceb8-4424-9404-64c98bd44b39-kube-api-access-gsgmx\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:48.066599 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.066513 5072 generic.go:356] "Generic (PLEG): container finished" podID="9a235824-ceb8-4424-9404-64c98bd44b39" containerID="d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4" exitCode=0 Oct 30 06:22:48.066757 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.066639 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" Oct 30 06:22:48.066757 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.066654 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" event={"ID":"9a235824-ceb8-4424-9404-64c98bd44b39","Type":"ContainerDied","Data":"d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4"} Oct 30 06:22:48.066757 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.066716 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl" event={"ID":"9a235824-ceb8-4424-9404-64c98bd44b39","Type":"ContainerDied","Data":"99a4aa5269498e0dad09ec8d4f6e664896a92d23d7c4ad7eaa7bdc4ce7d3d5a5"} Oct 30 06:22:48.066757 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.066739 5072 scope.go:122] "RemoveContainer" containerID="d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4" Oct 30 06:22:48.078566 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.078545 5072 scope.go:122] "RemoveContainer" containerID="d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4" Oct 30 06:22:48.078848 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:48.078832 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4\": container with ID starting with d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4 not found: ID does not exist" containerID="d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4" Oct 30 06:22:48.078897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.078858 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4"} err="failed to get container status \"d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4\": rpc error: code = NotFound desc = could not find container \"d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4\": container with ID starting with d6c674f69ed3278b0c6026c75eca65a9d52bafeaccf1d0fbd91a1933065760e4 not found: ID does not exist" Oct 30 06:22:48.122192 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.122158 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl"] Oct 30 06:22:48.135377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.135344 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fdb95-779447bb4d-5nktl"] Oct 30 06:22:48.522160 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:48.522124 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a235824-ceb8-4424-9404-64c98bd44b39" path="/var/lib/kubelet/pods/9a235824-ceb8-4424-9404-64c98bd44b39/volumes" Oct 30 06:22:50.762680 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:50.762645 5072 scope.go:122] "RemoveContainer" containerID="127f4f650bebfbe066e7f2b34a5ffdcce6544397adf5a5ba3de38df29fa80119" Oct 30 06:22:51.537246 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:51.537213 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s"] Oct 30 06:22:51.537525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:51.537500 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" podUID="b7183d70-926a-4483-a268-3dc50b09571d" containerName="maas-api" containerID="cri-o://6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2" gracePeriod=30 Oct 30 06:22:51.903598 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:51.903245 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:22:51.973403 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:51.973369 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7183d70-926a-4483-a268-3dc50b09571d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7183d70-926a-4483-a268-3dc50b09571d-openshift-service-ca\") pod \"b7183d70-926a-4483-a268-3dc50b09571d\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " Oct 30 06:22:51.973713 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:51.973689 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-metrics-certs\") pod \"b7183d70-926a-4483-a268-3dc50b09571d\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " Oct 30 06:22:51.973898 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:51.973883 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-maas-api-tls\") pod \"b7183d70-926a-4483-a268-3dc50b09571d\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " Oct 30 06:22:51.974059 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:51.974037 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7183d70-926a-4483-a268-3dc50b09571d-kube-api-access-4ngw4\" (UniqueName: \"kubernetes.io/projected/b7183d70-926a-4483-a268-3dc50b09571d-kube-api-access-4ngw4\") pod \"b7183d70-926a-4483-a268-3dc50b09571d\" (UID: \"b7183d70-926a-4483-a268-3dc50b09571d\") " Oct 30 06:22:52.014967 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.014916 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7183d70-926a-4483-a268-3dc50b09571d-openshift-service-ca" pod "b7183d70-926a-4483-a268-3dc50b09571d" (UID: "b7183d70-926a-4483-a268-3dc50b09571d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:52.022957 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.022906 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-maas-api-tls" pod "b7183d70-926a-4483-a268-3dc50b09571d" (UID: "b7183d70-926a-4483-a268-3dc50b09571d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:52.045522 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.045449 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-metrics-certs" pod "b7183d70-926a-4483-a268-3dc50b09571d" (UID: "b7183d70-926a-4483-a268-3dc50b09571d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:52.053566 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.050007 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7183d70-926a-4483-a268-3dc50b09571d-kube-api-access-4ngw4" pod "b7183d70-926a-4483-a268-3dc50b09571d" (UID: "b7183d70-926a-4483-a268-3dc50b09571d"). InnerVolumeSpecName "kube-api-access-4ngw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.075692 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.075661 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7183d70-926a-4483-a268-3dc50b09571d-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.075692 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.075691 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.075692 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.075700 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7183d70-926a-4483-a268-3dc50b09571d-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.075923 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.075708 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ngw4\" (UniqueName: \"kubernetes.io/projected/b7183d70-926a-4483-a268-3dc50b09571d-kube-api-access-4ngw4\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.095941 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.095909 5072 generic.go:356] "Generic (PLEG): container finished" podID="b7183d70-926a-4483-a268-3dc50b09571d" containerID="6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2" exitCode=0 Oct 30 06:22:52.096095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.095957 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" Oct 30 06:22:52.096095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.095989 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" event={"ID":"b7183d70-926a-4483-a268-3dc50b09571d","Type":"ContainerDied","Data":"6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2"} Oct 30 06:22:52.096095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.096044 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s" event={"ID":"b7183d70-926a-4483-a268-3dc50b09571d","Type":"ContainerDied","Data":"5b6f3776f73c69219b3794e9d5a3d4b26e9bd72af9e5f73abf2b599b1dcbdd62"} Oct 30 06:22:52.096095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.096068 5072 scope.go:122] "RemoveContainer" containerID="6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2" Oct 30 06:22:52.110212 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.110187 5072 scope.go:122] "RemoveContainer" containerID="6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2" Oct 30 06:22:52.110594 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:22:52.110561 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2\": container with ID starting with 6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2 not found: ID does not exist" containerID="6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2" Oct 30 06:22:52.110713 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.110601 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2"} err="failed to get container status \"6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2\": rpc error: code = NotFound desc = could not find container \"6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2\": container with ID starting with 6b8f8c94a0549376f9b3e1f40c5b747f797734f17f883b550a4158c7766a37c2 not found: ID does not exist" Oct 30 06:22:52.148285 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.148207 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s"] Oct 30 06:22:52.152130 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.152105 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6b8f87-9899fb47b-7589s"] Oct 30 06:22:52.519697 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:52.519610 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7183d70-926a-4483-a268-3dc50b09571d" path="/var/lib/kubelet/pods/b7183d70-926a-4483-a268-3dc50b09571d/volumes" Oct 30 06:22:56.058485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:56.057796 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq"] Oct 30 06:22:56.058485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:56.058044 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" containerID="cri-o://1f1312f304f7a75d3799a6e3b9a07d23bd7d3a10e821740fc5e13395d5c504b3" gracePeriod=30 Oct 30 06:22:59.512620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:59.512583 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h"] Oct 30 06:22:59.513149 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:22:59.512782 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" containerID="cri-o://c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99" gracePeriod=30 Oct 30 06:23:00.712581 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:00.712534 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:23:01.179772 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.179743 5072 generic.go:356] "Generic (PLEG): container finished" podID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerID="1f1312f304f7a75d3799a6e3b9a07d23bd7d3a10e821740fc5e13395d5c504b3" exitCode=0 Oct 30 06:23:01.179898 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.179818 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" event={"ID":"c54f773d-5f0c-4bde-a25f-5385934a95e1","Type":"ContainerDied","Data":"1f1312f304f7a75d3799a6e3b9a07d23bd7d3a10e821740fc5e13395d5c504b3"} Oct 30 06:23:01.405133 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.405107 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:23:01.557486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.557239 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-socket\") pod \"c54f773d-5f0c-4bde-a25f-5385934a95e1\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " Oct 30 06:23:01.557486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.557406 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-kube-api-access-8pg9x\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-kube-api-access-8pg9x\") pod \"c54f773d-5f0c-4bde-a25f-5385934a95e1\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " Oct 30 06:23:01.557486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.557483 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-podinfo\") pod \"c54f773d-5f0c-4bde-a25f-5385934a95e1\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " Oct 30 06:23:01.557841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.557527 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-certs\") pod \"c54f773d-5f0c-4bde-a25f-5385934a95e1\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " Oct 30 06:23:01.557841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.557566 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-credential-socket\") pod \"c54f773d-5f0c-4bde-a25f-5385934a95e1\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " Oct 30 06:23:01.557841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.557654 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-envoy\") pod \"c54f773d-5f0c-4bde-a25f-5385934a95e1\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " Oct 30 06:23:01.568133 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.561880 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-credential-socket" pod "c54f773d-5f0c-4bde-a25f-5385934a95e1" (UID: "c54f773d-5f0c-4bde-a25f-5385934a95e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.570593 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.569247 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-socket" pod "c54f773d-5f0c-4bde-a25f-5385934a95e1" (UID: "c54f773d-5f0c-4bde-a25f-5385934a95e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.570593 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.569413 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-certs" pod "c54f773d-5f0c-4bde-a25f-5385934a95e1" (UID: "c54f773d-5f0c-4bde-a25f-5385934a95e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.575644 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.575022 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-data\") pod \"c54f773d-5f0c-4bde-a25f-5385934a95e1\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " Oct 30 06:23:01.575644 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.575144 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-token\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-token\") pod \"c54f773d-5f0c-4bde-a25f-5385934a95e1\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " Oct 30 06:23:01.575644 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.575175 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54f773d-5f0c-4bde-a25f-5385934a95e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54f773d-5f0c-4bde-a25f-5385934a95e1-istiod-ca-cert\") pod \"c54f773d-5f0c-4bde-a25f-5385934a95e1\" (UID: \"c54f773d-5f0c-4bde-a25f-5385934a95e1\") " Oct 30 06:23:01.575927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.575800 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.575927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.575825 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.575927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.575839 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.578256 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.578206 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-data" pod "c54f773d-5f0c-4bde-a25f-5385934a95e1" (UID: "c54f773d-5f0c-4bde-a25f-5385934a95e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.578825 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.578783 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54f773d-5f0c-4bde-a25f-5385934a95e1-istiod-ca-cert" pod "c54f773d-5f0c-4bde-a25f-5385934a95e1" (UID: "c54f773d-5f0c-4bde-a25f-5385934a95e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:01.579268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.579209 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-envoy" pod "c54f773d-5f0c-4bde-a25f-5385934a95e1" (UID: "c54f773d-5f0c-4bde-a25f-5385934a95e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.581216 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.581170 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-podinfo" pod "c54f773d-5f0c-4bde-a25f-5385934a95e1" (UID: "c54f773d-5f0c-4bde-a25f-5385934a95e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:01.581573 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.581551 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-kube-api-access-8pg9x" pod "c54f773d-5f0c-4bde-a25f-5385934a95e1" (UID: "c54f773d-5f0c-4bde-a25f-5385934a95e1"). InnerVolumeSpecName "kube-api-access-8pg9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:01.582976 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.582917 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-token" pod "c54f773d-5f0c-4bde-a25f-5385934a95e1" (UID: "c54f773d-5f0c-4bde-a25f-5385934a95e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:01.677567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.677482 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pg9x\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-kube-api-access-8pg9x\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.677567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.677516 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.677567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.677530 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.677567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.677541 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.677567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.677552 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c54f773d-5f0c-4bde-a25f-5385934a95e1-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.677567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:01.677563 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c54f773d-5f0c-4bde-a25f-5385934a95e1-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.188668 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:02.188636 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" event={"ID":"c54f773d-5f0c-4bde-a25f-5385934a95e1","Type":"ContainerDied","Data":"41fdc5d7d38a8f35fd7ae1958d32cf937f58dc8821069f0f65be91594db33ed5"} Oct 30 06:23:02.189159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:02.188693 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq" Oct 30 06:23:02.189159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:02.188700 5072 scope.go:122] "RemoveContainer" containerID="1f1312f304f7a75d3799a6e3b9a07d23bd7d3a10e821740fc5e13395d5c504b3" Oct 30 06:23:02.245285 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:02.245247 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq"] Oct 30 06:23:02.256882 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:02.256854 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-v68hq"] Oct 30 06:23:02.522896 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:02.522814 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" path="/var/lib/kubelet/pods/c54f773d-5f0c-4bde-a25f-5385934a95e1/volumes" Oct 30 06:23:04.866812 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:04.866778 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:23:04.983277 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:04.983244 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5"] Oct 30 06:23:05.034894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.034860 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-podinfo\") pod \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " Oct 30 06:23:05.035099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.034911 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-socket\") pod \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " Oct 30 06:23:05.035099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.034972 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-certs\") pod \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " Oct 30 06:23:05.035099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035014 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-credential-socket\") pod \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " Oct 30 06:23:05.035099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035061 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-kube-api-access-b6d6j\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-kube-api-access-b6d6j\") pod \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " Oct 30 06:23:05.035099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035097 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istiod-ca-cert\") pod \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " Oct 30 06:23:05.035418 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035130 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-data\") pod \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " Oct 30 06:23:05.035418 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035162 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-token\") pod \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " Oct 30 06:23:05.035418 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035209 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-envoy\") pod \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\" (UID: \"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a\") " Oct 30 06:23:05.035418 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035285 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-socket" pod "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" (UID: "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:05.035418 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035405 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-credential-socket" pod "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" (UID: "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:05.035721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035419 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-data" pod "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" (UID: "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:05.035721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035539 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istiod-ca-cert" pod "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" (UID: "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:05.035840 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035740 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-certs" pod "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" (UID: "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:05.036032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035951 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.036032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035980 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.036032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.035993 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.036032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.036007 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.036032 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.036015 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.037709 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.037666 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-envoy" pod "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" (UID: "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:05.037709 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.037683 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-token" pod "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" (UID: "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:05.038098 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.038070 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-kube-api-access-b6d6j" pod "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" (UID: "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a"). InnerVolumeSpecName "kube-api-access-b6d6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:05.038380 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.038209 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-podinfo" pod "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" (UID: "dd1a82a3-3f78-4053-a9e6-3ca484c81f3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:05.136968 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.136932 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6d6j\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-kube-api-access-b6d6j\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.136968 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.136966 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.136968 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.136976 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.137258 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.136984 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.210858 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.210822 5072 generic.go:356] "Generic (PLEG): container finished" podID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerID="c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99" exitCode=0 Oct 30 06:23:05.211045 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.210882 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" Oct 30 06:23:05.211045 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.210907 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" event={"ID":"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a","Type":"ContainerDied","Data":"c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99"} Oct 30 06:23:05.211045 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.211008 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h" event={"ID":"dd1a82a3-3f78-4053-a9e6-3ca484c81f3a","Type":"ContainerDied","Data":"59b6aae026e7f9344070ed31157216369168a4d49e55980fd5f076237dda73a8"} Oct 30 06:23:05.211045 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.211037 5072 scope.go:122] "RemoveContainer" containerID="c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99" Oct 30 06:23:05.223206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.223185 5072 scope.go:122] "RemoveContainer" containerID="4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182" Oct 30 06:23:05.233571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.233547 5072 scope.go:122] "RemoveContainer" containerID="c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99" Oct 30 06:23:05.233941 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:05.233912 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99\": container with ID starting with c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99 not found: ID does not exist" containerID="c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99" Oct 30 06:23:05.234110 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.234079 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99"} err="failed to get container status \"c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99\": rpc error: code = NotFound desc = could not find container \"c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99\": container with ID starting with c50468fd2338e9a5fec5691a5ebaead22f71d559cbd496c2200c5d0e01fdfe99 not found: ID does not exist" Oct 30 06:23:05.234221 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.234204 5072 scope.go:122] "RemoveContainer" containerID="4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182" Oct 30 06:23:05.234891 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:05.234861 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182\": container with ID starting with 4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182 not found: ID does not exist" containerID="4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182" Oct 30 06:23:05.234985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.234899 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182"} err="failed to get container status \"4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182\": rpc error: code = NotFound desc = could not find container \"4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182\": container with ID starting with 4e0160f2c66e2e4c9f536d426fa73b8dafff20e10d61fb4574dae3008c7bd182 not found: ID does not exist" Oct 30 06:23:05.264009 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.263980 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h"] Oct 30 06:23:05.267846 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:05.267821 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-mk72h"] Oct 30 06:23:06.521376 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:06.521342 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" path="/var/lib/kubelet/pods/dd1a82a3-3f78-4053-a9e6-3ca484c81f3a/volumes" Oct 30 06:23:12.168551 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.168513 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk"] Oct 30 06:23:12.168965 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.168772 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" podUID="0359cfa7-f684-476e-a9e6-a2defd1feb2b" containerName="maas-api" containerID="cri-o://2de5f0e5086be1d4a2707b4699d43a28c205803de7998383367baea58de773b1" gracePeriod=30 Oct 30 06:23:12.304965 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.304936 5072 generic.go:356] "Generic (PLEG): container finished" podID="0359cfa7-f684-476e-a9e6-a2defd1feb2b" containerID="2de5f0e5086be1d4a2707b4699d43a28c205803de7998383367baea58de773b1" exitCode=0 Oct 30 06:23:12.305121 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.305021 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" event={"ID":"0359cfa7-f684-476e-a9e6-a2defd1feb2b","Type":"ContainerDied","Data":"2de5f0e5086be1d4a2707b4699d43a28c205803de7998383367baea58de773b1"} Oct 30 06:23:12.510169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.509346 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 06:23:12.636844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.636805 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0359cfa7-f684-476e-a9e6-a2defd1feb2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0359cfa7-f684-476e-a9e6-a2defd1feb2b-openshift-service-ca\") pod \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " Oct 30 06:23:12.637046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.636900 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0359cfa7-f684-476e-a9e6-a2defd1feb2b-kube-api-access-x7fl9\" (UniqueName: \"kubernetes.io/projected/0359cfa7-f684-476e-a9e6-a2defd1feb2b-kube-api-access-x7fl9\") pod \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " Oct 30 06:23:12.637046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.636937 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-maas-api-tls\") pod \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " Oct 30 06:23:12.637046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.636981 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-metrics-certs\") pod \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\" (UID: \"0359cfa7-f684-476e-a9e6-a2defd1feb2b\") " Oct 30 06:23:12.638368 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.638338 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0359cfa7-f684-476e-a9e6-a2defd1feb2b-openshift-service-ca" pod "0359cfa7-f684-476e-a9e6-a2defd1feb2b" (UID: "0359cfa7-f684-476e-a9e6-a2defd1feb2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:12.640605 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.640011 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-metrics-certs" pod "0359cfa7-f684-476e-a9e6-a2defd1feb2b" (UID: "0359cfa7-f684-476e-a9e6-a2defd1feb2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:12.640605 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.640189 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-maas-api-tls" pod "0359cfa7-f684-476e-a9e6-a2defd1feb2b" (UID: "0359cfa7-f684-476e-a9e6-a2defd1feb2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:12.641004 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.640982 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0359cfa7-f684-476e-a9e6-a2defd1feb2b-kube-api-access-x7fl9" pod "0359cfa7-f684-476e-a9e6-a2defd1feb2b" (UID: "0359cfa7-f684-476e-a9e6-a2defd1feb2b"). InnerVolumeSpecName "kube-api-access-x7fl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:12.739480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.738846 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0359cfa7-f684-476e-a9e6-a2defd1feb2b-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.739480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.738884 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7fl9\" (UniqueName: \"kubernetes.io/projected/0359cfa7-f684-476e-a9e6-a2defd1feb2b-kube-api-access-x7fl9\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.739480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.738897 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.739480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:12.738907 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0359cfa7-f684-476e-a9e6-a2defd1feb2b-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:13.313041 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:13.313014 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" Oct 30 06:23:13.313599 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:13.313046 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk" event={"ID":"0359cfa7-f684-476e-a9e6-a2defd1feb2b","Type":"ContainerDied","Data":"40996a75c3a58e8ef718edcac51f09d33a3896b0d2221ca981d4c0ce01224e89"} Oct 30 06:23:13.313599 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:13.313109 5072 scope.go:122] "RemoveContainer" containerID="2de5f0e5086be1d4a2707b4699d43a28c205803de7998383367baea58de773b1" Oct 30 06:23:13.364671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:13.364637 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk"] Oct 30 06:23:13.376828 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:13.376795 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3abe42-57c5bdcf75-crbnk"] Oct 30 06:23:14.518773 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:14.518743 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0359cfa7-f684-476e-a9e6-a2defd1feb2b" path="/var/lib/kubelet/pods/0359cfa7-f684-476e-a9e6-a2defd1feb2b/volumes" Oct 30 06:23:17.651853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:17.651818 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc"] Oct 30 06:23:17.652673 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:17.652639 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" podUID="af021843-9c8f-4ce6-8f08-858104de58dc" containerName="maas-api" containerID="cri-o://4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057" gracePeriod=30 Oct 30 06:23:17.807706 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:17.807668 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7"] Oct 30 06:23:17.807928 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:17.807903 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" podUID="788e07b5-328b-4d53-bb7d-c82b74d604d6" containerName="payload-processing" containerID="cri-o://df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd" gracePeriod=30 Oct 30 06:23:17.834049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:17.830155 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm"] Oct 30 06:23:17.834049 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:17.830521 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" podUID="74387953-1299-4d3b-8e25-995bd29ce315" containerName="payload-pre-processing" containerID="cri-o://d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a" gracePeriod=30 Oct 30 06:23:18.005236 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.005202 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:23:18.140490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.132871 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-maas-api-tls\") pod \"af021843-9c8f-4ce6-8f08-858104de58dc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " Oct 30 06:23:18.140490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.132945 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af021843-9c8f-4ce6-8f08-858104de58dc-kube-api-access-6gb98\" (UniqueName: \"kubernetes.io/projected/af021843-9c8f-4ce6-8f08-858104de58dc-kube-api-access-6gb98\") pod \"af021843-9c8f-4ce6-8f08-858104de58dc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " Oct 30 06:23:18.140490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.133030 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-metrics-certs\") pod \"af021843-9c8f-4ce6-8f08-858104de58dc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " Oct 30 06:23:18.140490 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.133150 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af021843-9c8f-4ce6-8f08-858104de58dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af021843-9c8f-4ce6-8f08-858104de58dc-openshift-service-ca\") pod \"af021843-9c8f-4ce6-8f08-858104de58dc\" (UID: \"af021843-9c8f-4ce6-8f08-858104de58dc\") " Oct 30 06:23:18.140861 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.140645 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af021843-9c8f-4ce6-8f08-858104de58dc-openshift-service-ca" pod "af021843-9c8f-4ce6-8f08-858104de58dc" (UID: "af021843-9c8f-4ce6-8f08-858104de58dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:18.194397 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.194130 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-maas-api-tls" pod "af021843-9c8f-4ce6-8f08-858104de58dc" (UID: "af021843-9c8f-4ce6-8f08-858104de58dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:18.194397 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.194303 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af021843-9c8f-4ce6-8f08-858104de58dc-kube-api-access-6gb98" pod "af021843-9c8f-4ce6-8f08-858104de58dc" (UID: "af021843-9c8f-4ce6-8f08-858104de58dc"). InnerVolumeSpecName "kube-api-access-6gb98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:18.208316 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.207494 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-metrics-certs" pod "af021843-9c8f-4ce6-8f08-858104de58dc" (UID: "af021843-9c8f-4ce6-8f08-858104de58dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:18.220723 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.220381 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:23:18.234911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.234807 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af021843-9c8f-4ce6-8f08-858104de58dc-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.234911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.234842 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.234911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.234860 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gb98\" (UniqueName: \"kubernetes.io/projected/af021843-9c8f-4ce6-8f08-858104de58dc-kube-api-access-6gb98\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.234911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.234873 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af021843-9c8f-4ce6-8f08-858104de58dc-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.293840 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.293818 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:23:18.337763 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.336672 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788e07b5-328b-4d53-bb7d-c82b74d604d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e07b5-328b-4d53-bb7d-c82b74d604d6-plugins-config-volume\") pod \"788e07b5-328b-4d53-bb7d-c82b74d604d6\" (UID: \"788e07b5-328b-4d53-bb7d-c82b74d604d6\") " Oct 30 06:23:18.337763 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.336977 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788e07b5-328b-4d53-bb7d-c82b74d604d6-kube-api-access-9pmsw\" (UniqueName: \"kubernetes.io/projected/788e07b5-328b-4d53-bb7d-c82b74d604d6-kube-api-access-9pmsw\") pod \"788e07b5-328b-4d53-bb7d-c82b74d604d6\" (UID: \"788e07b5-328b-4d53-bb7d-c82b74d604d6\") " Oct 30 06:23:18.338108 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.338066 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788e07b5-328b-4d53-bb7d-c82b74d604d6-plugins-config-volume" pod "788e07b5-328b-4d53-bb7d-c82b74d604d6" (UID: "788e07b5-328b-4d53-bb7d-c82b74d604d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:18.348510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.347290 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788e07b5-328b-4d53-bb7d-c82b74d604d6-kube-api-access-9pmsw" pod "788e07b5-328b-4d53-bb7d-c82b74d604d6" (UID: "788e07b5-328b-4d53-bb7d-c82b74d604d6"). InnerVolumeSpecName "kube-api-access-9pmsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:18.355904 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.355873 5072 generic.go:356] "Generic (PLEG): container finished" podID="af021843-9c8f-4ce6-8f08-858104de58dc" containerID="4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057" exitCode=0 Oct 30 06:23:18.356055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.355923 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" Oct 30 06:23:18.356055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.356004 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" event={"ID":"af021843-9c8f-4ce6-8f08-858104de58dc","Type":"ContainerDied","Data":"4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057"} Oct 30 06:23:18.356055 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.356043 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc" event={"ID":"af021843-9c8f-4ce6-8f08-858104de58dc","Type":"ContainerDied","Data":"b94d1f845ea3673e22ff572d2ec5f2d9d4f8a21ef6c21f00d0b6450068669ef5"} Oct 30 06:23:18.356225 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.356069 5072 scope.go:122] "RemoveContainer" containerID="4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057" Oct 30 06:23:18.359136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.358777 5072 generic.go:356] "Generic (PLEG): container finished" podID="74387953-1299-4d3b-8e25-995bd29ce315" containerID="d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a" exitCode=0 Oct 30 06:23:18.359136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.358825 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" event={"ID":"74387953-1299-4d3b-8e25-995bd29ce315","Type":"ContainerDied","Data":"d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a"} Oct 30 06:23:18.359136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.358892 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" event={"ID":"74387953-1299-4d3b-8e25-995bd29ce315","Type":"ContainerDied","Data":"761df4e8d1550130cb356dd9d1f71bb7182ebacd631ab8b60a75b7629a6ffc70"} Oct 30 06:23:18.359136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.358988 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm" Oct 30 06:23:18.360802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.360613 5072 generic.go:356] "Generic (PLEG): container finished" podID="788e07b5-328b-4d53-bb7d-c82b74d604d6" containerID="df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd" exitCode=0 Oct 30 06:23:18.360802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.360658 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" Oct 30 06:23:18.360802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.360731 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" event={"ID":"788e07b5-328b-4d53-bb7d-c82b74d604d6","Type":"ContainerDied","Data":"df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd"} Oct 30 06:23:18.360802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.360766 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7" event={"ID":"788e07b5-328b-4d53-bb7d-c82b74d604d6","Type":"ContainerDied","Data":"2b110e89b443f7dc85c5f21e97e21c6368ad1aa2d2ace19ca652be65cf986366"} Oct 30 06:23:18.371790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.371774 5072 scope.go:122] "RemoveContainer" containerID="4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057" Oct 30 06:23:18.372049 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:18.372024 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057\": container with ID starting with 4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057 not found: ID does not exist" containerID="4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057" Oct 30 06:23:18.372180 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.372061 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057"} err="failed to get container status \"4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057\": rpc error: code = NotFound desc = could not find container \"4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057\": container with ID starting with 4727ba7831632bea73ac951cad6d0af5acab8d4f988835f6a7b225336a82e057 not found: ID does not exist" Oct 30 06:23:18.372180 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.372084 5072 scope.go:122] "RemoveContainer" containerID="d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a" Oct 30 06:23:18.387036 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.387011 5072 scope.go:122] "RemoveContainer" containerID="d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a" Oct 30 06:23:18.387706 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:18.387673 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a\": container with ID starting with d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a not found: ID does not exist" containerID="d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a" Oct 30 06:23:18.387825 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.387713 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a"} err="failed to get container status \"d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a\": rpc error: code = NotFound desc = could not find container \"d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a\": container with ID starting with d24d1b8800e31fa399a33c4fd4981ee1969b6828f54a126d30ed3abd88c7940a not found: ID does not exist" Oct 30 06:23:18.387825 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.387736 5072 scope.go:122] "RemoveContainer" containerID="df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd" Oct 30 06:23:18.399003 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.398979 5072 scope.go:122] "RemoveContainer" containerID="df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd" Oct 30 06:23:18.399333 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:18.399312 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd\": container with ID starting with df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd not found: ID does not exist" containerID="df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd" Oct 30 06:23:18.399402 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.399347 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd"} err="failed to get container status \"df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd\": rpc error: code = NotFound desc = could not find container \"df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd\": container with ID starting with df9ec08e1dbe7fda1f2dbf626ac5c9b6bfbf624f73a5ea7f82f3ad139c34cdcd not found: ID does not exist" Oct 30 06:23:18.428713 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.428657 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc"] Oct 30 06:23:18.434657 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.434627 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-718730-59967bb8f6-gflqc"] Oct 30 06:23:18.438424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.438398 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74387953-1299-4d3b-8e25-995bd29ce315-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74387953-1299-4d3b-8e25-995bd29ce315-plugins-config-volume\") pod \"74387953-1299-4d3b-8e25-995bd29ce315\" (UID: \"74387953-1299-4d3b-8e25-995bd29ce315\") " Oct 30 06:23:18.438532 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.438505 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74387953-1299-4d3b-8e25-995bd29ce315-kube-api-access-6grdq\" (UniqueName: \"kubernetes.io/projected/74387953-1299-4d3b-8e25-995bd29ce315-kube-api-access-6grdq\") pod \"74387953-1299-4d3b-8e25-995bd29ce315\" (UID: \"74387953-1299-4d3b-8e25-995bd29ce315\") " Oct 30 06:23:18.438782 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.438760 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74387953-1299-4d3b-8e25-995bd29ce315-plugins-config-volume" pod "74387953-1299-4d3b-8e25-995bd29ce315" (UID: "74387953-1299-4d3b-8e25-995bd29ce315"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:18.439065 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.439048 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmsw\" (UniqueName: \"kubernetes.io/projected/788e07b5-328b-4d53-bb7d-c82b74d604d6-kube-api-access-9pmsw\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.439125 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.439071 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e07b5-328b-4d53-bb7d-c82b74d604d6-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.439125 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.439086 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74387953-1299-4d3b-8e25-995bd29ce315-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.440787 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.440765 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74387953-1299-4d3b-8e25-995bd29ce315-kube-api-access-6grdq" pod "74387953-1299-4d3b-8e25-995bd29ce315" (UID: "74387953-1299-4d3b-8e25-995bd29ce315"). InnerVolumeSpecName "kube-api-access-6grdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:18.482891 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.482826 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7"] Oct 30 06:23:18.487709 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.487687 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-718730-c86996588-f9ss7"] Oct 30 06:23:18.534703 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.534668 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788e07b5-328b-4d53-bb7d-c82b74d604d6" path="/var/lib/kubelet/pods/788e07b5-328b-4d53-bb7d-c82b74d604d6/volumes" Oct 30 06:23:18.535131 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.535118 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af021843-9c8f-4ce6-8f08-858104de58dc" path="/var/lib/kubelet/pods/af021843-9c8f-4ce6-8f08-858104de58dc/volumes" Oct 30 06:23:18.539851 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.539823 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6grdq\" (UniqueName: \"kubernetes.io/projected/74387953-1299-4d3b-8e25-995bd29ce315-kube-api-access-6grdq\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.699566 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.699530 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm"] Oct 30 06:23:18.702697 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:18.702667 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-75c58c6b99-djdqm"] Oct 30 06:23:20.006002 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.005946 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" containerID="cri-o://d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8" gracePeriod=45 Oct 30 06:23:20.196946 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.196915 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774"] Oct 30 06:23:20.197136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.197108 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" containerID="cri-o://0750afe02f8022dd0f7325d248463597a3df8b8eb791045466c05334639de385" gracePeriod=30 Oct 30 06:23:20.352406 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.352374 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 06:23:20.399377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.399326 5072 generic.go:356] "Generic (PLEG): container finished" podID="d7452308-ea76-448e-a885-300436798e92" containerID="d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8" exitCode=0 Oct 30 06:23:20.399581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.399395 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" event={"ID":"d7452308-ea76-448e-a885-300436798e92","Type":"ContainerDied","Data":"d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8"} Oct 30 06:23:20.399581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.399429 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" event={"ID":"d7452308-ea76-448e-a885-300436798e92","Type":"ContainerDied","Data":"69cd1e45af2d5e31479b9304c66053943ccf417cb259c9d5851bec0a01fb3343"} Oct 30 06:23:20.399581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.399449 5072 scope.go:122] "RemoveContainer" containerID="d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8" Oct 30 06:23:20.399581 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.399570 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" Oct 30 06:23:20.417881 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.417760 5072 scope.go:122] "RemoveContainer" containerID="d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8" Oct 30 06:23:20.418252 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:20.418175 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8\": container with ID starting with d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8 not found: ID does not exist" containerID="d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8" Oct 30 06:23:20.418252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.418213 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8"} err="failed to get container status \"d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8\": rpc error: code = NotFound desc = could not find container \"d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8\": container with ID starting with d103fdcb8d7f70e1c63e86cca234c42abcc2347712c131d159fd1ed517051fc8 not found: ID does not exist" Oct 30 06:23:20.485964 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.485930 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7452308-ea76-448e-a885-300436798e92-kube-api-access-r5tgb\" (UniqueName: \"kubernetes.io/projected/d7452308-ea76-448e-a885-300436798e92-kube-api-access-r5tgb\") pod \"d7452308-ea76-448e-a885-300436798e92\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " Oct 30 06:23:20.486165 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.486005 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-dshm\") pod \"d7452308-ea76-448e-a885-300436798e92\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " Oct 30 06:23:20.486165 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.486081 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-model-cache\") pod \"d7452308-ea76-448e-a885-300436798e92\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " Oct 30 06:23:20.486285 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.486167 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-home\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-home\") pod \"d7452308-ea76-448e-a885-300436798e92\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " Oct 30 06:23:20.486346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.486319 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7452308-ea76-448e-a885-300436798e92-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7452308-ea76-448e-a885-300436798e92-tls-certs\") pod \"d7452308-ea76-448e-a885-300436798e92\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " Oct 30 06:23:20.486404 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.486363 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-tmp-dir\") pod \"d7452308-ea76-448e-a885-300436798e92\" (UID: \"d7452308-ea76-448e-a885-300436798e92\") " Oct 30 06:23:20.486661 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.486633 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-model-cache" pod "d7452308-ea76-448e-a885-300436798e92" (UID: "d7452308-ea76-448e-a885-300436798e92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.486860 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.486842 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.486860 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.486841 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-tmp-dir" pod "d7452308-ea76-448e-a885-300436798e92" (UID: "d7452308-ea76-448e-a885-300436798e92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.487016 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.486998 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-home" pod "d7452308-ea76-448e-a885-300436798e92" (UID: "d7452308-ea76-448e-a885-300436798e92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.499979 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.499949 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-dshm" pod "d7452308-ea76-448e-a885-300436798e92" (UID: "d7452308-ea76-448e-a885-300436798e92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.500572 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.500543 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7452308-ea76-448e-a885-300436798e92-kube-api-access-r5tgb" pod "d7452308-ea76-448e-a885-300436798e92" (UID: "d7452308-ea76-448e-a885-300436798e92"). InnerVolumeSpecName "kube-api-access-r5tgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:20.502241 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.502207 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7452308-ea76-448e-a885-300436798e92-tls-certs" pod "d7452308-ea76-448e-a885-300436798e92" (UID: "d7452308-ea76-448e-a885-300436798e92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:20.539035 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.538954 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74387953-1299-4d3b-8e25-995bd29ce315" path="/var/lib/kubelet/pods/74387953-1299-4d3b-8e25-995bd29ce315/volumes" Oct 30 06:23:20.588095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.588060 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7452308-ea76-448e-a885-300436798e92-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.588095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.588088 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.588095 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.588100 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tgb\" (UniqueName: \"kubernetes.io/projected/d7452308-ea76-448e-a885-300436798e92-kube-api-access-r5tgb\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.588365 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.588114 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.588365 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.588139 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7452308-ea76-448e-a885-300436798e92-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.745037 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.744996 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5"] Oct 30 06:23:20.747633 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:20.747608 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5"] Oct 30 06:23:21.332519 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:21.332447 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-w3-fffaa8-kserve-74d88cf5f8-566j5" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:23:22.520734 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:22.520702 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7452308-ea76-448e-a885-300436798e92" path="/var/lib/kubelet/pods/d7452308-ea76-448e-a885-300436798e92/volumes" Oct 30 06:23:25.349500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.342409 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm"] Oct 30 06:23:25.349500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.342707 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" containerID="cri-o://db633516ed6556654a8babcaa65d56636d58e4a4e7f64950b8aba02bb6e73a43" gracePeriod=30 Oct 30 06:23:25.530344 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.530320 5072 generic.go:356] "Generic (PLEG): container finished" podID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerID="0750afe02f8022dd0f7325d248463597a3df8b8eb791045466c05334639de385" exitCode=0 Oct 30 06:23:25.530476 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.530388 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" event={"ID":"c30d89ef-de1f-493d-b956-70ffc9d4943c","Type":"ContainerDied","Data":"0750afe02f8022dd0f7325d248463597a3df8b8eb791045466c05334639de385"} Oct 30 06:23:25.559398 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.559367 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:23:25.619893 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.619856 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-data\") pod \"c30d89ef-de1f-493d-b956-70ffc9d4943c\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " Oct 30 06:23:25.620098 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.619910 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-envoy\") pod \"c30d89ef-de1f-493d-b956-70ffc9d4943c\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " Oct 30 06:23:25.620098 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.619991 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-kube-api-access-xnrht\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-kube-api-access-xnrht\") pod \"c30d89ef-de1f-493d-b956-70ffc9d4943c\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " Oct 30 06:23:25.620098 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620041 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-certs\") pod \"c30d89ef-de1f-493d-b956-70ffc9d4943c\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " Oct 30 06:23:25.620281 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620096 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-credential-socket\") pod \"c30d89ef-de1f-493d-b956-70ffc9d4943c\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " Oct 30 06:23:25.620281 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620160 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-data" pod "c30d89ef-de1f-493d-b956-70ffc9d4943c" (UID: "c30d89ef-de1f-493d-b956-70ffc9d4943c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.620281 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620192 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c30d89ef-de1f-493d-b956-70ffc9d4943c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30d89ef-de1f-493d-b956-70ffc9d4943c-istiod-ca-cert\") pod \"c30d89ef-de1f-493d-b956-70ffc9d4943c\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " Oct 30 06:23:25.620281 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620236 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-token\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-token\") pod \"c30d89ef-de1f-493d-b956-70ffc9d4943c\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620342 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-podinfo\") pod \"c30d89ef-de1f-493d-b956-70ffc9d4943c\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620383 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-credential-socket" pod "c30d89ef-de1f-493d-b956-70ffc9d4943c" (UID: "c30d89ef-de1f-493d-b956-70ffc9d4943c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620387 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-socket\") pod \"c30d89ef-de1f-493d-b956-70ffc9d4943c\" (UID: \"c30d89ef-de1f-493d-b956-70ffc9d4943c\") " Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620548 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-certs" pod "c30d89ef-de1f-493d-b956-70ffc9d4943c" (UID: "c30d89ef-de1f-493d-b956-70ffc9d4943c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.620605 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-socket" pod "c30d89ef-de1f-493d-b956-70ffc9d4943c" (UID: "c30d89ef-de1f-493d-b956-70ffc9d4943c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.621306 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.621504 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30d89ef-de1f-493d-b956-70ffc9d4943c-istiod-ca-cert" pod "c30d89ef-de1f-493d-b956-70ffc9d4943c" (UID: "c30d89ef-de1f-493d-b956-70ffc9d4943c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.621524 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.621544 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.621844 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.621558 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.623621 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.623594 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-kube-api-access-xnrht" pod "c30d89ef-de1f-493d-b956-70ffc9d4943c" (UID: "c30d89ef-de1f-493d-b956-70ffc9d4943c"). InnerVolumeSpecName "kube-api-access-xnrht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:25.625910 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.625885 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-envoy" pod "c30d89ef-de1f-493d-b956-70ffc9d4943c" (UID: "c30d89ef-de1f-493d-b956-70ffc9d4943c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.626338 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.626315 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-podinfo" pod "c30d89ef-de1f-493d-b956-70ffc9d4943c" (UID: "c30d89ef-de1f-493d-b956-70ffc9d4943c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:25.626428 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.626354 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-token" pod "c30d89ef-de1f-493d-b956-70ffc9d4943c" (UID: "c30d89ef-de1f-493d-b956-70ffc9d4943c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:25.722566 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.722525 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.722566 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.722564 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.722786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.722579 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnrht\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-kube-api-access-xnrht\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.722786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.722594 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30d89ef-de1f-493d-b956-70ffc9d4943c-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.722786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:25.722609 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c30d89ef-de1f-493d-b956-70ffc9d4943c-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:26.538155 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:26.538120 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" event={"ID":"c30d89ef-de1f-493d-b956-70ffc9d4943c","Type":"ContainerDied","Data":"03fdf0e379b9a4a8d662386d27cd18c9ab31a1782422efb9cdfe4d380c4d9390"} Oct 30 06:23:26.538604 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:26.538165 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774" Oct 30 06:23:26.538604 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:26.538186 5072 scope.go:122] "RemoveContainer" containerID="0750afe02f8022dd0f7325d248463597a3df8b8eb791045466c05334639de385" Oct 30 06:23:26.602380 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:26.602350 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774"] Oct 30 06:23:26.614777 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:26.614744 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-62774"] Oct 30 06:23:26.832786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:26.825288 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg"] Oct 30 06:23:26.832786 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:26.825734 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" podUID="d74d09be-6408-4b4f-bdbf-f36c2953566f" containerName="maas-api" containerID="cri-o://141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318" gracePeriod=30 Oct 30 06:23:27.138327 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.138300 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:23:27.147707 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.147681 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-metrics-certs\") pod \"d74d09be-6408-4b4f-bdbf-f36c2953566f\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " Oct 30 06:23:27.147863 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.147746 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74d09be-6408-4b4f-bdbf-f36c2953566f-kube-api-access-gjj4b\" (UniqueName: \"kubernetes.io/projected/d74d09be-6408-4b4f-bdbf-f36c2953566f-kube-api-access-gjj4b\") pod \"d74d09be-6408-4b4f-bdbf-f36c2953566f\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " Oct 30 06:23:27.147863 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.147771 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74d09be-6408-4b4f-bdbf-f36c2953566f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d09be-6408-4b4f-bdbf-f36c2953566f-openshift-service-ca\") pod \"d74d09be-6408-4b4f-bdbf-f36c2953566f\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " Oct 30 06:23:27.147863 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.147830 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-maas-api-tls\") pod \"d74d09be-6408-4b4f-bdbf-f36c2953566f\" (UID: \"d74d09be-6408-4b4f-bdbf-f36c2953566f\") " Oct 30 06:23:27.148154 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.148126 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74d09be-6408-4b4f-bdbf-f36c2953566f-openshift-service-ca" pod "d74d09be-6408-4b4f-bdbf-f36c2953566f" (UID: "d74d09be-6408-4b4f-bdbf-f36c2953566f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:27.148330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.148312 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74d09be-6408-4b4f-bdbf-f36c2953566f-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.150100 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.150077 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-metrics-certs" pod "d74d09be-6408-4b4f-bdbf-f36c2953566f" (UID: "d74d09be-6408-4b4f-bdbf-f36c2953566f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:27.150207 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.150109 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-maas-api-tls" pod "d74d09be-6408-4b4f-bdbf-f36c2953566f" (UID: "d74d09be-6408-4b4f-bdbf-f36c2953566f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:27.150401 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.150381 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74d09be-6408-4b4f-bdbf-f36c2953566f-kube-api-access-gjj4b" pod "d74d09be-6408-4b4f-bdbf-f36c2953566f" (UID: "d74d09be-6408-4b4f-bdbf-f36c2953566f"). InnerVolumeSpecName "kube-api-access-gjj4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:27.249943 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.249901 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.249943 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.249941 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjj4b\" (UniqueName: \"kubernetes.io/projected/d74d09be-6408-4b4f-bdbf-f36c2953566f-kube-api-access-gjj4b\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.250161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.249959 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74d09be-6408-4b4f-bdbf-f36c2953566f-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.547590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.547494 5072 generic.go:356] "Generic (PLEG): container finished" podID="d74d09be-6408-4b4f-bdbf-f36c2953566f" containerID="141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318" exitCode=0 Oct 30 06:23:27.547590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.547533 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" Oct 30 06:23:27.548181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.547585 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" event={"ID":"d74d09be-6408-4b4f-bdbf-f36c2953566f","Type":"ContainerDied","Data":"141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318"} Oct 30 06:23:27.548181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.547661 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg" event={"ID":"d74d09be-6408-4b4f-bdbf-f36c2953566f","Type":"ContainerDied","Data":"ee682e6a3adb70168247e451520ae03f69f656dd40bb90f851cdaa105d1c9883"} Oct 30 06:23:27.548181 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.547695 5072 scope.go:122] "RemoveContainer" containerID="141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318" Oct 30 06:23:27.559627 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.558677 5072 scope.go:122] "RemoveContainer" containerID="141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318" Oct 30 06:23:27.559627 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:27.558990 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318\": container with ID starting with 141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318 not found: ID does not exist" containerID="141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318" Oct 30 06:23:27.559627 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.559043 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318"} err="failed to get container status \"141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318\": rpc error: code = NotFound desc = could not find container \"141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318\": container with ID starting with 141d2d4df3f0e3aa6246faf111f0bde4b04c259d6e85b5dfaba7bd4f4dda3318 not found: ID does not exist" Oct 30 06:23:27.605378 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.605346 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg"] Oct 30 06:23:27.617111 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:27.617072 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-720e9c-76c4dc97b5-nqwwg"] Oct 30 06:23:28.520327 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:28.520294 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" path="/var/lib/kubelet/pods/c30d89ef-de1f-493d-b956-70ffc9d4943c/volumes" Oct 30 06:23:28.520879 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:28.520854 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74d09be-6408-4b4f-bdbf-f36c2953566f" path="/var/lib/kubelet/pods/d74d09be-6408-4b4f-bdbf-f36c2953566f/volumes" Oct 30 06:23:28.746262 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:28.746226 5072 patch_prober.go:29] interesting pod/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:23:28.746688 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:28.746294 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:23:30.580555 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.577905 5072 generic.go:356] "Generic (PLEG): container finished" podID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerID="db633516ed6556654a8babcaa65d56636d58e4a4e7f64950b8aba02bb6e73a43" exitCode=0 Oct 30 06:23:30.580555 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.577986 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" event={"ID":"3bf8ac2d-724c-4309-aaba-de0952fc9802","Type":"ContainerDied","Data":"db633516ed6556654a8babcaa65d56636d58e4a4e7f64950b8aba02bb6e73a43"} Oct 30 06:23:30.580555 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.578044 5072 scope.go:122] "RemoveContainer" containerID="6ca4efaf2d9c50e7b0393d48c3803f210bc380f94e8375273aea3246fc41d33a" Oct 30 06:23:30.657982 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.657951 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:23:30.691762 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.691732 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-envoy\") pod \"3bf8ac2d-724c-4309-aaba-de0952fc9802\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " Oct 30 06:23:30.691933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.691787 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-data\") pod \"3bf8ac2d-724c-4309-aaba-de0952fc9802\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " Oct 30 06:23:30.691933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.691810 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-certs\") pod \"3bf8ac2d-724c-4309-aaba-de0952fc9802\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " Oct 30 06:23:30.691933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.691833 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-kube-api-access-rl4kl\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-kube-api-access-rl4kl\") pod \"3bf8ac2d-724c-4309-aaba-de0952fc9802\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " Oct 30 06:23:30.691933 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.691859 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-credential-socket\") pod \"3bf8ac2d-724c-4309-aaba-de0952fc9802\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " Oct 30 06:23:30.692252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692100 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf8ac2d-724c-4309-aaba-de0952fc9802-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8ac2d-724c-4309-aaba-de0952fc9802-istiod-ca-cert\") pod \"3bf8ac2d-724c-4309-aaba-de0952fc9802\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " Oct 30 06:23:30.692252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692125 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-certs" pod "3bf8ac2d-724c-4309-aaba-de0952fc9802" (UID: "3bf8ac2d-724c-4309-aaba-de0952fc9802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:30.692252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692161 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-podinfo\") pod \"3bf8ac2d-724c-4309-aaba-de0952fc9802\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " Oct 30 06:23:30.692252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692217 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-credential-socket" pod "3bf8ac2d-724c-4309-aaba-de0952fc9802" (UID: "3bf8ac2d-724c-4309-aaba-de0952fc9802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:30.692252 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692230 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-data" pod "3bf8ac2d-724c-4309-aaba-de0952fc9802" (UID: "3bf8ac2d-724c-4309-aaba-de0952fc9802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:30.692570 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692264 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-socket\") pod \"3bf8ac2d-724c-4309-aaba-de0952fc9802\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " Oct 30 06:23:30.692570 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692357 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-token\") pod \"3bf8ac2d-724c-4309-aaba-de0952fc9802\" (UID: \"3bf8ac2d-724c-4309-aaba-de0952fc9802\") " Oct 30 06:23:30.692570 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692521 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-socket" pod "3bf8ac2d-724c-4309-aaba-de0952fc9802" (UID: "3bf8ac2d-724c-4309-aaba-de0952fc9802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:30.692904 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692734 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf8ac2d-724c-4309-aaba-de0952fc9802-istiod-ca-cert" pod "3bf8ac2d-724c-4309-aaba-de0952fc9802" (UID: "3bf8ac2d-724c-4309-aaba-de0952fc9802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:30.692969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692930 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:30.692969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692953 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:30.693086 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692969 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:30.693086 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692984 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf8ac2d-724c-4309-aaba-de0952fc9802-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:30.693086 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.692998 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:30.694552 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.694526 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-kube-api-access-rl4kl" pod "3bf8ac2d-724c-4309-aaba-de0952fc9802" (UID: "3bf8ac2d-724c-4309-aaba-de0952fc9802"). InnerVolumeSpecName "kube-api-access-rl4kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:30.694663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.694608 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-envoy" pod "3bf8ac2d-724c-4309-aaba-de0952fc9802" (UID: "3bf8ac2d-724c-4309-aaba-de0952fc9802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:30.694832 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.694810 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-token" pod "3bf8ac2d-724c-4309-aaba-de0952fc9802" (UID: "3bf8ac2d-724c-4309-aaba-de0952fc9802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:30.695510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.695492 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-podinfo" pod "3bf8ac2d-724c-4309-aaba-de0952fc9802" (UID: "3bf8ac2d-724c-4309-aaba-de0952fc9802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:30.713272 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:30.713208 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:23:30.794013 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.793978 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:30.794013 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.794007 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl4kl\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-kube-api-access-rl4kl\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:30.794229 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.794020 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:30.794229 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:30.794034 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf8ac2d-724c-4309-aaba-de0952fc9802-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:31.626527 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:31.624164 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" event={"ID":"3bf8ac2d-724c-4309-aaba-de0952fc9802","Type":"ContainerDied","Data":"8a6d0b220399f39a1520db875140dded33be5c15d28fc5ac73f65146dd1bfbbe"} Oct 30 06:23:31.626527 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:31.624245 5072 scope.go:122] "RemoveContainer" containerID="db633516ed6556654a8babcaa65d56636d58e4a4e7f64950b8aba02bb6e73a43" Oct 30 06:23:31.626527 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:31.624373 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm" Oct 30 06:23:31.677605 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:31.677578 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm"] Oct 30 06:23:31.689624 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:31.689596 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-718730-openshift-default-7dc8849dc7-fswsm"] Oct 30 06:23:32.521012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:32.520976 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" path="/var/lib/kubelet/pods/3bf8ac2d-724c-4309-aaba-de0952fc9802/volumes" Oct 30 06:23:40.656493 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:40.656435 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb"] Oct 30 06:23:40.656948 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:40.656700 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" podUID="28026c62-1c09-45d4-9d3e-810c67b7117c" containerName="maas-api" containerID="cri-o://9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9" gracePeriod=30 Oct 30 06:23:40.995317 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:40.995280 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 06:23:41.145039 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.145009 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28026c62-1c09-45d4-9d3e-810c67b7117c-kube-api-access-gm7rw\" (UniqueName: \"kubernetes.io/projected/28026c62-1c09-45d4-9d3e-810c67b7117c-kube-api-access-gm7rw\") pod \"28026c62-1c09-45d4-9d3e-810c67b7117c\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " Oct 30 06:23:41.145217 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.145120 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-metrics-certs\") pod \"28026c62-1c09-45d4-9d3e-810c67b7117c\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " Oct 30 06:23:41.145298 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.145280 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28026c62-1c09-45d4-9d3e-810c67b7117c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28026c62-1c09-45d4-9d3e-810c67b7117c-openshift-service-ca\") pod \"28026c62-1c09-45d4-9d3e-810c67b7117c\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " Oct 30 06:23:41.145345 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.145316 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls\") pod \"28026c62-1c09-45d4-9d3e-810c67b7117c\" (UID: \"28026c62-1c09-45d4-9d3e-810c67b7117c\") " Oct 30 06:23:41.145736 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.145702 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28026c62-1c09-45d4-9d3e-810c67b7117c-openshift-service-ca" pod "28026c62-1c09-45d4-9d3e-810c67b7117c" (UID: "28026c62-1c09-45d4-9d3e-810c67b7117c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:41.146027 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.145999 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28026c62-1c09-45d4-9d3e-810c67b7117c-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.147420 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.147393 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-metrics-certs" pod "28026c62-1c09-45d4-9d3e-810c67b7117c" (UID: "28026c62-1c09-45d4-9d3e-810c67b7117c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:41.147554 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.147518 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28026c62-1c09-45d4-9d3e-810c67b7117c-kube-api-access-gm7rw" pod "28026c62-1c09-45d4-9d3e-810c67b7117c" (UID: "28026c62-1c09-45d4-9d3e-810c67b7117c"). InnerVolumeSpecName "kube-api-access-gm7rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:41.147603 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.147545 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls" pod "28026c62-1c09-45d4-9d3e-810c67b7117c" (UID: "28026c62-1c09-45d4-9d3e-810c67b7117c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:41.246927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.246853 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.246927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.246878 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28026c62-1c09-45d4-9d3e-810c67b7117c-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.246927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.246888 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm7rw\" (UniqueName: \"kubernetes.io/projected/28026c62-1c09-45d4-9d3e-810c67b7117c-kube-api-access-gm7rw\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.705720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.705686 5072 generic.go:356] "Generic (PLEG): container finished" podID="28026c62-1c09-45d4-9d3e-810c67b7117c" containerID="9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9" exitCode=0 Oct 30 06:23:41.706234 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.705737 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" Oct 30 06:23:41.706234 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.705766 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" event={"ID":"28026c62-1c09-45d4-9d3e-810c67b7117c","Type":"ContainerDied","Data":"9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9"} Oct 30 06:23:41.706234 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.705827 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb" event={"ID":"28026c62-1c09-45d4-9d3e-810c67b7117c","Type":"ContainerDied","Data":"220eee863ee6cf6d19074c06ad94c3dc3bbb0a2343eae2c9fcf700c5427d978f"} Oct 30 06:23:41.706234 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.705854 5072 scope.go:122] "RemoveContainer" containerID="9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9" Oct 30 06:23:41.717609 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.717583 5072 scope.go:122] "RemoveContainer" containerID="9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9" Oct 30 06:23:41.717896 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:41.717875 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9\": container with ID starting with 9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9 not found: ID does not exist" containerID="9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9" Oct 30 06:23:41.717972 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.717907 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9"} err="failed to get container status \"9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9\": rpc error: code = NotFound desc = could not find container \"9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9\": container with ID starting with 9dad3adf25eb44765969e5083b7acac1071599206ecbc113acc9146662b04bd9 not found: ID does not exist" Oct 30 06:23:41.759136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.759105 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb"] Oct 30 06:23:41.769745 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:41.769720 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-fffaa8-78485c5b75-6kznb"] Oct 30 06:23:42.519725 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:42.519698 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28026c62-1c09-45d4-9d3e-810c67b7117c" path="/var/lib/kubelet/pods/28026c62-1c09-45d4-9d3e-810c67b7117c/volumes" Oct 30 06:23:43.369974 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.369935 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx"] Oct 30 06:23:43.370507 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.370251 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" podUID="5ea933c4-0f19-489d-a553-98d2da67114a" containerName="maas-api" containerID="cri-o://8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114" gracePeriod=30 Oct 30 06:23:43.698285 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.698253 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:23:43.725687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.725652 5072 generic.go:356] "Generic (PLEG): container finished" podID="5ea933c4-0f19-489d-a553-98d2da67114a" containerID="8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114" exitCode=0 Oct 30 06:23:43.725871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.725710 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" Oct 30 06:23:43.725871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.725745 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" event={"ID":"5ea933c4-0f19-489d-a553-98d2da67114a","Type":"ContainerDied","Data":"8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114"} Oct 30 06:23:43.725871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.725807 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx" event={"ID":"5ea933c4-0f19-489d-a553-98d2da67114a","Type":"ContainerDied","Data":"4ac993547e160e2daf6fc89594baa020eb188f6d201218ad5d47cf3b156dc226"} Oct 30 06:23:43.725871 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.725840 5072 scope.go:122] "RemoveContainer" containerID="8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114" Oct 30 06:23:43.736091 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.736065 5072 scope.go:122] "RemoveContainer" containerID="8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114" Oct 30 06:23:43.736385 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:43.736363 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114\": container with ID starting with 8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114 not found: ID does not exist" containerID="8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114" Oct 30 06:23:43.736484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.736395 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114"} err="failed to get container status \"8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114\": rpc error: code = NotFound desc = could not find container \"8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114\": container with ID starting with 8fd4a7073e207d5b8222ee88c9e9f2b8c33b6cdfc59e9d05d999d9d313eeb114 not found: ID does not exist" Oct 30 06:23:43.790170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.790123 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea933c4-0f19-489d-a553-98d2da67114a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea933c4-0f19-489d-a553-98d2da67114a-openshift-service-ca\") pod \"5ea933c4-0f19-489d-a553-98d2da67114a\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " Oct 30 06:23:43.790351 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.790226 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls\") pod \"5ea933c4-0f19-489d-a553-98d2da67114a\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " Oct 30 06:23:43.790351 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.790279 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea933c4-0f19-489d-a553-98d2da67114a-kube-api-access-lldfx\" (UniqueName: \"kubernetes.io/projected/5ea933c4-0f19-489d-a553-98d2da67114a-kube-api-access-lldfx\") pod \"5ea933c4-0f19-489d-a553-98d2da67114a\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " Oct 30 06:23:43.790558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.790367 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-metrics-certs\") pod \"5ea933c4-0f19-489d-a553-98d2da67114a\" (UID: \"5ea933c4-0f19-489d-a553-98d2da67114a\") " Oct 30 06:23:43.790658 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.790632 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea933c4-0f19-489d-a553-98d2da67114a-openshift-service-ca" pod "5ea933c4-0f19-489d-a553-98d2da67114a" (UID: "5ea933c4-0f19-489d-a553-98d2da67114a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:43.790950 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.790923 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea933c4-0f19-489d-a553-98d2da67114a-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:43.793349 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.793323 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-metrics-certs" pod "5ea933c4-0f19-489d-a553-98d2da67114a" (UID: "5ea933c4-0f19-489d-a553-98d2da67114a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:43.793560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.793448 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea933c4-0f19-489d-a553-98d2da67114a-kube-api-access-lldfx" pod "5ea933c4-0f19-489d-a553-98d2da67114a" (UID: "5ea933c4-0f19-489d-a553-98d2da67114a"). InnerVolumeSpecName "kube-api-access-lldfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:43.793659 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.793606 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls" pod "5ea933c4-0f19-489d-a553-98d2da67114a" (UID: "5ea933c4-0f19-489d-a553-98d2da67114a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:43.893280 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.892339 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:43.893280 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.892382 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lldfx\" (UniqueName: \"kubernetes.io/projected/5ea933c4-0f19-489d-a553-98d2da67114a-kube-api-access-lldfx\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:43.893280 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:43.892398 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea933c4-0f19-489d-a553-98d2da67114a-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.075817 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:44.075772 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx"] Oct 30 06:23:44.079655 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:44.079630 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-c2be21-5895668c8f-mlwlx"] Oct 30 06:23:44.520679 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:44.520644 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea933c4-0f19-489d-a553-98d2da67114a" path="/var/lib/kubelet/pods/5ea933c4-0f19-489d-a553-98d2da67114a/volumes" Oct 30 06:23:49.264146 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:49.264112 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm"] Oct 30 06:23:49.264657 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:49.264383 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" containerID="cri-o://66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909" gracePeriod=30 Oct 30 06:23:49.909206 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:49.909169 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq"] Oct 30 06:23:49.916530 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:49.910056 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerName="istio-proxy" containerID="cri-o://41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406" gracePeriod=30 Oct 30 06:23:51.798495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:51.785789 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj"] Oct 30 06:23:51.798495 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:51.786960 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" containerID="cri-o://c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c" gracePeriod=30 Oct 30 06:23:52.168789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:52.168747 5072 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:23:52.168969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:52.168820 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:23:54.418802 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.418760 5072 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 06:23:54.419316 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.418832 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 06:23:54.532710 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.532644 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 06:23:54.622976 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.622942 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-data\") pod \"76b8e9c3-0956-46e3-bcb7-02225a62c460\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " Oct 30 06:23:54.623176 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623100 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-credential-socket\") pod \"76b8e9c3-0956-46e3-bcb7-02225a62c460\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " Oct 30 06:23:54.623176 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623140 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-kube-api-access-q2l9p\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-kube-api-access-q2l9p\") pod \"76b8e9c3-0956-46e3-bcb7-02225a62c460\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " Oct 30 06:23:54.623300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623214 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-socket\") pod \"76b8e9c3-0956-46e3-bcb7-02225a62c460\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " Oct 30 06:23:54.623300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623235 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-data" pod "76b8e9c3-0956-46e3-bcb7-02225a62c460" (UID: "76b8e9c3-0956-46e3-bcb7-02225a62c460"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.623300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623278 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-certs\") pod \"76b8e9c3-0956-46e3-bcb7-02225a62c460\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " Oct 30 06:23:54.623500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623321 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b8e9c3-0956-46e3-bcb7-02225a62c460-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8e9c3-0956-46e3-bcb7-02225a62c460-istiod-ca-cert\") pod \"76b8e9c3-0956-46e3-bcb7-02225a62c460\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " Oct 30 06:23:54.623500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623411 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-podinfo\") pod \"76b8e9c3-0956-46e3-bcb7-02225a62c460\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " Oct 30 06:23:54.623500 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623453 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-token\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-token\") pod \"76b8e9c3-0956-46e3-bcb7-02225a62c460\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " Oct 30 06:23:54.623705 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623539 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-envoy\") pod \"76b8e9c3-0956-46e3-bcb7-02225a62c460\" (UID: \"76b8e9c3-0956-46e3-bcb7-02225a62c460\") " Oct 30 06:23:54.623705 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623549 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-credential-socket" pod "76b8e9c3-0956-46e3-bcb7-02225a62c460" (UID: "76b8e9c3-0956-46e3-bcb7-02225a62c460"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.623705 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623616 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-socket" pod "76b8e9c3-0956-46e3-bcb7-02225a62c460" (UID: "76b8e9c3-0956-46e3-bcb7-02225a62c460"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.624311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.623939 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b8e9c3-0956-46e3-bcb7-02225a62c460-istiod-ca-cert" pod "76b8e9c3-0956-46e3-bcb7-02225a62c460" (UID: "76b8e9c3-0956-46e3-bcb7-02225a62c460"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:54.624311 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.624218 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-certs" pod "76b8e9c3-0956-46e3-bcb7-02225a62c460" (UID: "76b8e9c3-0956-46e3-bcb7-02225a62c460"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.624423 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.624371 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.624423 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.624390 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8e9c3-0956-46e3-bcb7-02225a62c460-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.624423 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.624402 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.624423 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.624414 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.624831 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.624428 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.626646 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.626614 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-podinfo" pod "76b8e9c3-0956-46e3-bcb7-02225a62c460" (UID: "76b8e9c3-0956-46e3-bcb7-02225a62c460"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:54.627189 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.627167 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-token" pod "76b8e9c3-0956-46e3-bcb7-02225a62c460" (UID: "76b8e9c3-0956-46e3-bcb7-02225a62c460"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:54.627300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.627203 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-kube-api-access-q2l9p" pod "76b8e9c3-0956-46e3-bcb7-02225a62c460" (UID: "76b8e9c3-0956-46e3-bcb7-02225a62c460"). InnerVolumeSpecName "kube-api-access-q2l9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:54.627367 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.627330 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-envoy" pod "76b8e9c3-0956-46e3-bcb7-02225a62c460" (UID: "76b8e9c3-0956-46e3-bcb7-02225a62c460"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.726099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.726057 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.726099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.726091 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.726099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.726100 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8e9c3-0956-46e3-bcb7-02225a62c460-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.726099 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.726108 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2l9p\" (UniqueName: \"kubernetes.io/projected/76b8e9c3-0956-46e3-bcb7-02225a62c460-kube-api-access-q2l9p\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.820268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.820182 5072 generic.go:356] "Generic (PLEG): container finished" podID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerID="66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909" exitCode=0 Oct 30 06:23:54.820268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.820254 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" Oct 30 06:23:54.820537 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.820271 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" event={"ID":"76b8e9c3-0956-46e3-bcb7-02225a62c460","Type":"ContainerDied","Data":"66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909"} Oct 30 06:23:54.820537 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.820331 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm" event={"ID":"76b8e9c3-0956-46e3-bcb7-02225a62c460","Type":"ContainerDied","Data":"ebd7c809af2643c36ead0ad1d16acf81caee58bd6e42f4cc031f95e687fc0b3a"} Oct 30 06:23:54.820537 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.820342 5072 scope.go:122] "RemoveContainer" containerID="66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909" Oct 30 06:23:54.831235 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.831176 5072 scope.go:122] "RemoveContainer" containerID="66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909" Oct 30 06:23:54.831558 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:54.831529 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909\": container with ID starting with 66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909 not found: ID does not exist" containerID="66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909" Oct 30 06:23:54.831639 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.831572 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909"} err="failed to get container status \"66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909\": rpc error: code = NotFound desc = could not find container \"66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909\": container with ID starting with 66b7971a09df0d3d2c49e76e29fcba2f533f483d4d505efb5981120aff372909 not found: ID does not exist" Oct 30 06:23:54.872053 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.872024 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm"] Oct 30 06:23:54.885944 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:54.885915 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-49glm"] Oct 30 06:23:55.172976 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.172950 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:23:55.231994 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.231959 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-podinfo\") pod \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " Oct 30 06:23:55.232184 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.232020 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-socket\") pod \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " Oct 30 06:23:55.232184 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.232134 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-envoy\") pod \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " Oct 30 06:23:55.232336 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.232251 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istiod-ca-cert\") pod \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " Oct 30 06:23:55.232336 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.232300 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-certs\") pod \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " Oct 30 06:23:55.232506 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.232340 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-kube-api-access-cv8jw\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-kube-api-access-cv8jw\") pod \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " Oct 30 06:23:55.232506 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.232382 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-credential-socket\") pod \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " Oct 30 06:23:55.232506 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.232439 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-token\") pod \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " Oct 30 06:23:55.232696 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.232521 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-data\") pod \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\" (UID: \"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0\") " Oct 30 06:23:55.232884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.232858 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istiod-ca-cert" pod "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" (UID: "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:55.233082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.233056 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-socket" pod "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" (UID: "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.233186 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.233080 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.233186 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.233089 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-data" pod "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" (UID: "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.233315 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.233290 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-credential-socket" pod "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" (UID: "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.233315 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.233302 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-certs" pod "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" (UID: "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.235300 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.235270 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-token" pod "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" (UID: "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:55.235787 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.235766 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-kube-api-access-cv8jw" pod "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" (UID: "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0"). InnerVolumeSpecName "kube-api-access-cv8jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:55.235872 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.235797 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-envoy" pod "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" (UID: "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.235872 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.235799 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-podinfo" pod "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" (UID: "5cb930a6-978c-413f-b1fb-6b7ab90d5fc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:55.334618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.334550 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.334618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.334579 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.334618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.334587 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.334618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.334597 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.334618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.334606 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.334618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.334615 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.334618 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.334623 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv8jw\" (UniqueName: \"kubernetes.io/projected/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-kube-api-access-cv8jw\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.334963 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.334632 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.828278 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.828246 5072 generic.go:356] "Generic (PLEG): container finished" podID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerID="41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406" exitCode=0 Oct 30 06:23:55.828776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.828300 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" Oct 30 06:23:55.828776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.828332 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" event={"ID":"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0","Type":"ContainerDied","Data":"41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406"} Oct 30 06:23:55.828776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.828395 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq" event={"ID":"5cb930a6-978c-413f-b1fb-6b7ab90d5fc0","Type":"ContainerDied","Data":"dda6def25d63fafa5d655329780bd2a440c533626c3c3267bef1d69447bce393"} Oct 30 06:23:55.828776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.828431 5072 scope.go:122] "RemoveContainer" containerID="41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406" Oct 30 06:23:55.852592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.852548 5072 scope.go:122] "RemoveContainer" containerID="41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406" Oct 30 06:23:55.853025 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:55.852993 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406\": container with ID starting with 41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406 not found: ID does not exist" containerID="41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406" Oct 30 06:23:55.853152 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.853034 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406"} err="failed to get container status \"41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406\": rpc error: code = NotFound desc = could not find container \"41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406\": container with ID starting with 41c5b0d1ecf7659a548984475dbbc204e06dbd0378027f94fde8adea25f7f406 not found: ID does not exist" Oct 30 06:23:55.887815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.887783 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq"] Oct 30 06:23:55.909384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:55.909350 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-z8jgq"] Oct 30 06:23:56.519910 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:56.519875 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" path="/var/lib/kubelet/pods/5cb930a6-978c-413f-b1fb-6b7ab90d5fc0/volumes" Oct 30 06:23:56.520360 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:56.520338 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" path="/var/lib/kubelet/pods/76b8e9c3-0956-46e3-bcb7-02225a62c460/volumes" Oct 30 06:23:57.092286 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.092255 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:23:57.160682 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.160651 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-envoy\") pod \"b959f379-69f3-4be5-ac96-8fd4bc2211be\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " Oct 30 06:23:57.160881 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.160718 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-credential-socket\") pod \"b959f379-69f3-4be5-ac96-8fd4bc2211be\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " Oct 30 06:23:57.160881 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.160766 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-podinfo\") pod \"b959f379-69f3-4be5-ac96-8fd4bc2211be\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " Oct 30 06:23:57.160881 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.160824 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-data\") pod \"b959f379-69f3-4be5-ac96-8fd4bc2211be\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " Oct 30 06:23:57.160881 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.160869 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-socket\") pod \"b959f379-69f3-4be5-ac96-8fd4bc2211be\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " Oct 30 06:23:57.161139 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.160911 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-kube-api-access-f562x\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-kube-api-access-f562x\") pod \"b959f379-69f3-4be5-ac96-8fd4bc2211be\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " Oct 30 06:23:57.161139 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.160938 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-certs\") pod \"b959f379-69f3-4be5-ac96-8fd4bc2211be\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " Oct 30 06:23:57.161139 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.160972 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-token\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-token\") pod \"b959f379-69f3-4be5-ac96-8fd4bc2211be\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " Oct 30 06:23:57.161139 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.161002 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b959f379-69f3-4be5-ac96-8fd4bc2211be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b959f379-69f3-4be5-ac96-8fd4bc2211be-istiod-ca-cert\") pod \"b959f379-69f3-4be5-ac96-8fd4bc2211be\" (UID: \"b959f379-69f3-4be5-ac96-8fd4bc2211be\") " Oct 30 06:23:57.161139 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.161085 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-credential-socket" pod "b959f379-69f3-4be5-ac96-8fd4bc2211be" (UID: "b959f379-69f3-4be5-ac96-8fd4bc2211be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:57.161424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.161160 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-data" pod "b959f379-69f3-4be5-ac96-8fd4bc2211be" (UID: "b959f379-69f3-4be5-ac96-8fd4bc2211be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:57.161424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.161272 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-certs" pod "b959f379-69f3-4be5-ac96-8fd4bc2211be" (UID: "b959f379-69f3-4be5-ac96-8fd4bc2211be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:57.161580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.161492 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.161580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.161514 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.161580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.161529 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.161580 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.161492 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-socket" pod "b959f379-69f3-4be5-ac96-8fd4bc2211be" (UID: "b959f379-69f3-4be5-ac96-8fd4bc2211be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:57.162155 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.162129 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b959f379-69f3-4be5-ac96-8fd4bc2211be-istiod-ca-cert" pod "b959f379-69f3-4be5-ac96-8fd4bc2211be" (UID: "b959f379-69f3-4be5-ac96-8fd4bc2211be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:57.163586 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.163558 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-podinfo" pod "b959f379-69f3-4be5-ac96-8fd4bc2211be" (UID: "b959f379-69f3-4be5-ac96-8fd4bc2211be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:57.163806 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.163786 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-token" pod "b959f379-69f3-4be5-ac96-8fd4bc2211be" (UID: "b959f379-69f3-4be5-ac96-8fd4bc2211be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:57.165528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.165502 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-envoy" pod "b959f379-69f3-4be5-ac96-8fd4bc2211be" (UID: "b959f379-69f3-4be5-ac96-8fd4bc2211be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:57.166920 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.166897 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-kube-api-access-f562x" pod "b959f379-69f3-4be5-ac96-8fd4bc2211be" (UID: "b959f379-69f3-4be5-ac96-8fd4bc2211be"). InnerVolumeSpecName "kube-api-access-f562x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:57.262754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.262721 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.262754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.262751 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f562x\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-kube-api-access-f562x\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.262754 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.262762 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.263019 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.262771 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b959f379-69f3-4be5-ac96-8fd4bc2211be-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.263019 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.262779 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.263019 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.262787 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b959f379-69f3-4be5-ac96-8fd4bc2211be-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:23:57.844868 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.844834 5072 generic.go:356] "Generic (PLEG): container finished" podID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerID="c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c" exitCode=0 Oct 30 06:23:57.845091 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.844894 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" Oct 30 06:23:57.845091 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.844915 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" event={"ID":"b959f379-69f3-4be5-ac96-8fd4bc2211be","Type":"ContainerDied","Data":"c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c"} Oct 30 06:23:57.845091 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.844966 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj" event={"ID":"b959f379-69f3-4be5-ac96-8fd4bc2211be","Type":"ContainerDied","Data":"e7eb5c8e3ab56e064acc3aad928a8bec2749e6917a1a03cf91b49b35e7129092"} Oct 30 06:23:57.845091 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.844986 5072 scope.go:122] "RemoveContainer" containerID="c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c" Oct 30 06:23:57.855158 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.855132 5072 scope.go:122] "RemoveContainer" containerID="8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d" Oct 30 06:23:57.864847 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.864830 5072 scope.go:122] "RemoveContainer" containerID="c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c" Oct 30 06:23:57.865129 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:57.865110 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c\": container with ID starting with c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c not found: ID does not exist" containerID="c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c" Oct 30 06:23:57.865178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.865139 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c"} err="failed to get container status \"c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c\": rpc error: code = NotFound desc = could not find container \"c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c\": container with ID starting with c0b6e6b8760afafbfd4438edaff6efdd3da296c3a1d6c8d7865bbdce90a9519c not found: ID does not exist" Oct 30 06:23:57.865178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.865156 5072 scope.go:122] "RemoveContainer" containerID="8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d" Oct 30 06:23:57.865415 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:23:57.865393 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d\": container with ID starting with 8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d not found: ID does not exist" containerID="8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d" Oct 30 06:23:57.865508 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.865425 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d"} err="failed to get container status \"8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d\": rpc error: code = NotFound desc = could not find container \"8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d\": container with ID starting with 8726548d2ab31b0518ad86cb1546ab0b45506475129fc03a1baa84181d56550d not found: ID does not exist" Oct 30 06:23:57.897046 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.897013 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj"] Oct 30 06:23:57.909210 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:57.909179 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c2be21-openshift-default-76cd6899f-k46bj"] Oct 30 06:23:58.519243 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:23:58.519206 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" path="/var/lib/kubelet/pods/b959f379-69f3-4be5-ac96-8fd4bc2211be/volumes" Oct 30 06:24:00.715033 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:00.714990 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:24:01.341636 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:01.341356 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh"] Oct 30 06:24:01.341859 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:01.341645 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" containerID="cri-o://001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f" gracePeriod=30 Oct 30 06:24:02.707880 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:02.707847 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx"] Oct 30 06:24:04.677215 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:04.677182 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk"] Oct 30 06:24:04.677766 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:04.677440 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" podUID="b04f54d9-44d0-4fae-af50-c0c6e20df84c" containerName="maas-api" containerID="cri-o://68347b6b4d79923bcf563bdcb09a0aa44e6bceea5e3492b834176d5f40a9499f" gracePeriod=30 Oct 30 06:24:04.896349 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:04.896140 5072 generic.go:356] "Generic (PLEG): container finished" podID="b04f54d9-44d0-4fae-af50-c0c6e20df84c" containerID="68347b6b4d79923bcf563bdcb09a0aa44e6bceea5e3492b834176d5f40a9499f" exitCode=0 Oct 30 06:24:04.896349 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:04.896286 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" event={"ID":"b04f54d9-44d0-4fae-af50-c0c6e20df84c","Type":"ContainerDied","Data":"68347b6b4d79923bcf563bdcb09a0aa44e6bceea5e3492b834176d5f40a9499f"} Oct 30 06:24:04.971391 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:04.971363 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:24:05.056169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.054693 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls\") pod \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " Oct 30 06:24:05.056169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.055823 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-metrics-certs\") pod \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " Oct 30 06:24:05.056169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.055876 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04f54d9-44d0-4fae-af50-c0c6e20df84c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04f54d9-44d0-4fae-af50-c0c6e20df84c-openshift-service-ca\") pod \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " Oct 30 06:24:05.057487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.056867 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04f54d9-44d0-4fae-af50-c0c6e20df84c-kube-api-access-bp29z\" (UniqueName: \"kubernetes.io/projected/b04f54d9-44d0-4fae-af50-c0c6e20df84c-kube-api-access-bp29z\") pod \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\" (UID: \"b04f54d9-44d0-4fae-af50-c0c6e20df84c\") " Oct 30 06:24:05.057487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.057111 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04f54d9-44d0-4fae-af50-c0c6e20df84c-openshift-service-ca" pod "b04f54d9-44d0-4fae-af50-c0c6e20df84c" (UID: "b04f54d9-44d0-4fae-af50-c0c6e20df84c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:05.057702 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.057591 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04f54d9-44d0-4fae-af50-c0c6e20df84c-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:05.068976 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.068938 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04f54d9-44d0-4fae-af50-c0c6e20df84c-kube-api-access-bp29z" pod "b04f54d9-44d0-4fae-af50-c0c6e20df84c" (UID: "b04f54d9-44d0-4fae-af50-c0c6e20df84c"). InnerVolumeSpecName "kube-api-access-bp29z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:05.075637 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.075576 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-metrics-certs" pod "b04f54d9-44d0-4fae-af50-c0c6e20df84c" (UID: "b04f54d9-44d0-4fae-af50-c0c6e20df84c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:05.075927 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.075899 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls" pod "b04f54d9-44d0-4fae-af50-c0c6e20df84c" (UID: "b04f54d9-44d0-4fae-af50-c0c6e20df84c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:05.158961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.158924 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:05.158961 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.158965 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b04f54d9-44d0-4fae-af50-c0c6e20df84c-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:05.159161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.158981 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp29z\" (UniqueName: \"kubernetes.io/projected/b04f54d9-44d0-4fae-af50-c0c6e20df84c-kube-api-access-bp29z\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:05.904865 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.904768 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" Oct 30 06:24:05.904865 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.904794 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk" event={"ID":"b04f54d9-44d0-4fae-af50-c0c6e20df84c","Type":"ContainerDied","Data":"172add3365d2b2c8e3c59ece9241aa4dbf8a6b2a081f9f4221ed9e9354893700"} Oct 30 06:24:05.905476 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.904871 5072 scope.go:122] "RemoveContainer" containerID="68347b6b4d79923bcf563bdcb09a0aa44e6bceea5e3492b834176d5f40a9499f" Oct 30 06:24:05.959157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.959119 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk"] Oct 30 06:24:05.971134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:05.971099 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2f1e4c-86c66c986b-8cdhk"] Oct 30 06:24:06.520857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.520825 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04f54d9-44d0-4fae-af50-c0c6e20df84c" path="/var/lib/kubelet/pods/b04f54d9-44d0-4fae-af50-c0c6e20df84c/volumes" Oct 30 06:24:06.653620 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.653580 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:24:06.779172 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779137 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-podinfo\") pod \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " Oct 30 06:24:06.779172 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779178 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-envoy\") pod \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " Oct 30 06:24:06.779426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779223 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-token\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-token\") pod \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " Oct 30 06:24:06.779426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779247 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-kube-api-access-sdj4n\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-kube-api-access-sdj4n\") pod \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " Oct 30 06:24:06.779426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779266 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-credential-socket\") pod \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " Oct 30 06:24:06.779426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779284 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istiod-ca-cert\") pod \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " Oct 30 06:24:06.779426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779397 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-data\") pod \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " Oct 30 06:24:06.779426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779425 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-socket\") pod \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " Oct 30 06:24:06.779782 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779477 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-certs\") pod \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\" (UID: \"60b49a18-306a-4cbe-8fe3-1295c43ca4f3\") " Oct 30 06:24:06.779782 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779753 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-data" pod "60b49a18-306a-4cbe-8fe3-1295c43ca4f3" (UID: "60b49a18-306a-4cbe-8fe3-1295c43ca4f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:06.779782 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779774 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istiod-ca-cert" pod "60b49a18-306a-4cbe-8fe3-1295c43ca4f3" (UID: "60b49a18-306a-4cbe-8fe3-1295c43ca4f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:06.779978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779957 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-credential-socket" pod "60b49a18-306a-4cbe-8fe3-1295c43ca4f3" (UID: "60b49a18-306a-4cbe-8fe3-1295c43ca4f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:06.780114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.779976 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-certs" pod "60b49a18-306a-4cbe-8fe3-1295c43ca4f3" (UID: "60b49a18-306a-4cbe-8fe3-1295c43ca4f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:06.780114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.780015 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-socket" pod "60b49a18-306a-4cbe-8fe3-1295c43ca4f3" (UID: "60b49a18-306a-4cbe-8fe3-1295c43ca4f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:06.780114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.780099 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.780114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.780112 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.780114 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.780125 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.780592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.780139 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.780592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.780151 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.781796 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.781772 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-envoy" pod "60b49a18-306a-4cbe-8fe3-1295c43ca4f3" (UID: "60b49a18-306a-4cbe-8fe3-1295c43ca4f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:06.781906 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.781772 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-podinfo" pod "60b49a18-306a-4cbe-8fe3-1295c43ca4f3" (UID: "60b49a18-306a-4cbe-8fe3-1295c43ca4f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:06.782100 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.782081 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-kube-api-access-sdj4n" pod "60b49a18-306a-4cbe-8fe3-1295c43ca4f3" (UID: "60b49a18-306a-4cbe-8fe3-1295c43ca4f3"). InnerVolumeSpecName "kube-api-access-sdj4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:06.782258 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.782236 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-token" pod "60b49a18-306a-4cbe-8fe3-1295c43ca4f3" (UID: "60b49a18-306a-4cbe-8fe3-1295c43ca4f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:06.881729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.881642 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.881729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.881673 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.881729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.881686 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.881729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.881699 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdj4n\" (UniqueName: \"kubernetes.io/projected/60b49a18-306a-4cbe-8fe3-1295c43ca4f3-kube-api-access-sdj4n\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.914424 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.914387 5072 generic.go:356] "Generic (PLEG): container finished" podID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerID="001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f" exitCode=0 Oct 30 06:24:06.915531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.914472 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" Oct 30 06:24:06.915531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.914544 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" event={"ID":"60b49a18-306a-4cbe-8fe3-1295c43ca4f3","Type":"ContainerDied","Data":"001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f"} Oct 30 06:24:06.915531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.914593 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh" event={"ID":"60b49a18-306a-4cbe-8fe3-1295c43ca4f3","Type":"ContainerDied","Data":"3a6c87b63c8ffee926b7ee1ec3563c7b90b0c25b0edb8f27e3f8431fe01fa47f"} Oct 30 06:24:06.915531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.914621 5072 scope.go:122] "RemoveContainer" containerID="001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f" Oct 30 06:24:06.929382 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.929270 5072 scope.go:122] "RemoveContainer" containerID="1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c" Oct 30 06:24:06.939298 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.939278 5072 scope.go:122] "RemoveContainer" containerID="001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f" Oct 30 06:24:06.939655 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:06.939629 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f\": container with ID starting with 001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f not found: ID does not exist" containerID="001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f" Oct 30 06:24:06.939792 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.939676 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f"} err="failed to get container status \"001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f\": rpc error: code = NotFound desc = could not find container \"001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f\": container with ID starting with 001a76f36b84bbe8fb63d2c1109c6c7ab20f957393287ac2e3a8e2b57ecde06f not found: ID does not exist" Oct 30 06:24:06.939792 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.939702 5072 scope.go:122] "RemoveContainer" containerID="1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c" Oct 30 06:24:06.940013 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:06.939990 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c\": container with ID starting with 1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c not found: ID does not exist" containerID="1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c" Oct 30 06:24:06.940091 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.940024 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c"} err="failed to get container status \"1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c\": rpc error: code = NotFound desc = could not find container \"1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c\": container with ID starting with 1aaaa2b5efce55caa41c6e7def668da9f02558b84c2ad1248f1afd9682ce590c not found: ID does not exist" Oct 30 06:24:06.971244 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.970964 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh"] Oct 30 06:24:06.975110 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:06.975073 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-xh2kh"] Oct 30 06:24:07.709130 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:07.709080 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" podUID="c0bd7978-6f64-456c-b403-108050d494e2" containerName="payload-pre-processing" containerID="cri-o://7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc" gracePeriod=25 Oct 30 06:24:08.519727 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:08.519696 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" path="/var/lib/kubelet/pods/60b49a18-306a-4cbe-8fe3-1295c43ca4f3/volumes" Oct 30 06:24:12.151077 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:12.151030 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" podUID="c0bd7978-6f64-456c-b403-108050d494e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:13.140294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:13.140251 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw"] Oct 30 06:24:13.140858 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:13.140804 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" containerID="cri-o://62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e" gracePeriod=30 Oct 30 06:24:13.916011 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:13.915973 5072 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:24:13.916406 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:13.916036 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:24:18.439698 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.439602 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:24:18.483739 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.483701 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr"] Oct 30 06:24:18.483984 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.483962 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" podUID="17588e96-befd-440e-aca9-e8582a61fcdf" containerName="maas-api" containerID="cri-o://f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8" gracePeriod=30 Oct 30 06:24:18.608084 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.608042 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c"] Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.617007 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-token\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-token\") pod \"cd71ff23-7006-4545-aeb3-087a3310a0bb\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.617092 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd71ff23-7006-4545-aeb3-087a3310a0bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd71ff23-7006-4545-aeb3-087a3310a0bb-istiod-ca-cert\") pod \"cd71ff23-7006-4545-aeb3-087a3310a0bb\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.617126 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-socket\") pod \"cd71ff23-7006-4545-aeb3-087a3310a0bb\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.617193 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-envoy\") pod \"cd71ff23-7006-4545-aeb3-087a3310a0bb\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.617246 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-kube-api-access-jlvw4\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-kube-api-access-jlvw4\") pod \"cd71ff23-7006-4545-aeb3-087a3310a0bb\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.617290 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-podinfo\") pod \"cd71ff23-7006-4545-aeb3-087a3310a0bb\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.617376 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-certs\") pod \"cd71ff23-7006-4545-aeb3-087a3310a0bb\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.617411 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-credential-socket\") pod \"cd71ff23-7006-4545-aeb3-087a3310a0bb\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.617443 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-data\") pod \"cd71ff23-7006-4545-aeb3-087a3310a0bb\" (UID: \"cd71ff23-7006-4545-aeb3-087a3310a0bb\") " Oct 30 06:24:18.621332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.620897 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-data" pod "cd71ff23-7006-4545-aeb3-087a3310a0bb" (UID: "cd71ff23-7006-4545-aeb3-087a3310a0bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.623004 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.622932 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd71ff23-7006-4545-aeb3-087a3310a0bb-istiod-ca-cert" pod "cd71ff23-7006-4545-aeb3-087a3310a0bb" (UID: "cd71ff23-7006-4545-aeb3-087a3310a0bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:18.625087 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.624396 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-credential-socket" pod "cd71ff23-7006-4545-aeb3-087a3310a0bb" (UID: "cd71ff23-7006-4545-aeb3-087a3310a0bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.625087 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.624722 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-socket" pod "cd71ff23-7006-4545-aeb3-087a3310a0bb" (UID: "cd71ff23-7006-4545-aeb3-087a3310a0bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.625087 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.625046 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-certs" pod "cd71ff23-7006-4545-aeb3-087a3310a0bb" (UID: "cd71ff23-7006-4545-aeb3-087a3310a0bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.643717 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.643671 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-envoy" pod "cd71ff23-7006-4545-aeb3-087a3310a0bb" (UID: "cd71ff23-7006-4545-aeb3-087a3310a0bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.649328 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.649227 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-kube-api-access-jlvw4" pod "cd71ff23-7006-4545-aeb3-087a3310a0bb" (UID: "cd71ff23-7006-4545-aeb3-087a3310a0bb"). InnerVolumeSpecName "kube-api-access-jlvw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:18.649328 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.649248 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-token" pod "cd71ff23-7006-4545-aeb3-087a3310a0bb" (UID: "cd71ff23-7006-4545-aeb3-087a3310a0bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:18.649328 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.649298 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-podinfo" pod "cd71ff23-7006-4545-aeb3-087a3310a0bb" (UID: "cd71ff23-7006-4545-aeb3-087a3310a0bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:18.726671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.721219 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.726671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.721261 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlvw4\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-kube-api-access-jlvw4\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.726671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.721278 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.726671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.721294 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.726671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.721308 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.726671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.721321 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.726671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.721334 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd71ff23-7006-4545-aeb3-087a3310a0bb-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.726671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.721348 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd71ff23-7006-4545-aeb3-087a3310a0bb-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.726671 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.721363 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd71ff23-7006-4545-aeb3-087a3310a0bb-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.803583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.803558 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 06:24:18.923702 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.923650 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-maas-api-tls\") pod \"17588e96-befd-440e-aca9-e8582a61fcdf\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " Oct 30 06:24:18.923926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.923735 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-metrics-certs\") pod \"17588e96-befd-440e-aca9-e8582a61fcdf\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " Oct 30 06:24:18.923926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.923886 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17588e96-befd-440e-aca9-e8582a61fcdf-kube-api-access-s2c6q\" (UniqueName: \"kubernetes.io/projected/17588e96-befd-440e-aca9-e8582a61fcdf-kube-api-access-s2c6q\") pod \"17588e96-befd-440e-aca9-e8582a61fcdf\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " Oct 30 06:24:18.923926 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.923920 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17588e96-befd-440e-aca9-e8582a61fcdf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17588e96-befd-440e-aca9-e8582a61fcdf-openshift-service-ca\") pod \"17588e96-befd-440e-aca9-e8582a61fcdf\" (UID: \"17588e96-befd-440e-aca9-e8582a61fcdf\") " Oct 30 06:24:18.924708 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.924679 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17588e96-befd-440e-aca9-e8582a61fcdf-openshift-service-ca" pod "17588e96-befd-440e-aca9-e8582a61fcdf" (UID: "17588e96-befd-440e-aca9-e8582a61fcdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:18.929976 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.929915 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-maas-api-tls" pod "17588e96-befd-440e-aca9-e8582a61fcdf" (UID: "17588e96-befd-440e-aca9-e8582a61fcdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:18.958198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.958103 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17588e96-befd-440e-aca9-e8582a61fcdf-kube-api-access-s2c6q" pod "17588e96-befd-440e-aca9-e8582a61fcdf" (UID: "17588e96-befd-440e-aca9-e8582a61fcdf"). InnerVolumeSpecName "kube-api-access-s2c6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:18.961110 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.961080 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-metrics-certs" pod "17588e96-befd-440e-aca9-e8582a61fcdf" (UID: "17588e96-befd-440e-aca9-e8582a61fcdf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:18.998392 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.998359 5072 generic.go:356] "Generic (PLEG): container finished" podID="17588e96-befd-440e-aca9-e8582a61fcdf" containerID="f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8" exitCode=0 Oct 30 06:24:18.998613 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.998410 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" Oct 30 06:24:18.998613 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.998438 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" event={"ID":"17588e96-befd-440e-aca9-e8582a61fcdf","Type":"ContainerDied","Data":"f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8"} Oct 30 06:24:18.998613 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.998525 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr" event={"ID":"17588e96-befd-440e-aca9-e8582a61fcdf","Type":"ContainerDied","Data":"030f0ba566efca380da7cfdda7f7463f5ecd2e8fdf84cad53a7beb2f2141989f"} Oct 30 06:24:18.998613 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:18.998551 5072 scope.go:122] "RemoveContainer" containerID="f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8" Oct 30 06:24:19.000951 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.000926 5072 generic.go:356] "Generic (PLEG): container finished" podID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerID="62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e" exitCode=0 Oct 30 06:24:19.001108 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.000978 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" event={"ID":"cd71ff23-7006-4545-aeb3-087a3310a0bb","Type":"ContainerDied","Data":"62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e"} Oct 30 06:24:19.001108 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.001012 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" event={"ID":"cd71ff23-7006-4545-aeb3-087a3310a0bb","Type":"ContainerDied","Data":"5e2311fa8edbdffc2c10722e129baf22757bc750a2c84faabb64f85b91f020c8"} Oct 30 06:24:19.001108 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.001073 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw" Oct 30 06:24:19.025443 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.025341 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2c6q\" (UniqueName: \"kubernetes.io/projected/17588e96-befd-440e-aca9-e8582a61fcdf-kube-api-access-s2c6q\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:19.025443 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.025377 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17588e96-befd-440e-aca9-e8582a61fcdf-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:19.025443 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.025393 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:19.025443 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.025408 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17588e96-befd-440e-aca9-e8582a61fcdf-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:19.042030 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.042002 5072 scope.go:122] "RemoveContainer" containerID="f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8" Oct 30 06:24:19.042657 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:19.042599 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8\": container with ID starting with f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8 not found: ID does not exist" containerID="f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8" Oct 30 06:24:19.042946 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.042825 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8"} err="failed to get container status \"f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8\": rpc error: code = NotFound desc = could not find container \"f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8\": container with ID starting with f5b7959488829278d60794ba0e6de4822da21695ba01114f5255b550d9965ee8 not found: ID does not exist" Oct 30 06:24:19.042946 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.042859 5072 scope.go:122] "RemoveContainer" containerID="62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e" Oct 30 06:24:19.054669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.054629 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr"] Oct 30 06:24:19.059177 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.059149 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1d8c2f-69b94f8949-25zpr"] Oct 30 06:24:19.065295 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.065275 5072 scope.go:122] "RemoveContainer" containerID="8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad" Oct 30 06:24:19.076640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.076621 5072 scope.go:122] "RemoveContainer" containerID="62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e" Oct 30 06:24:19.076943 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:19.076910 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e\": container with ID starting with 62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e not found: ID does not exist" containerID="62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e" Oct 30 06:24:19.077135 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.076954 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e"} err="failed to get container status \"62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e\": rpc error: code = NotFound desc = could not find container \"62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e\": container with ID starting with 62dbdd0eab5ebd31bc322842e10afc49088342dfcce1d5b746d1c1959264578e not found: ID does not exist" Oct 30 06:24:19.077135 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.076981 5072 scope.go:122] "RemoveContainer" containerID="8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad" Oct 30 06:24:19.077310 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:19.077289 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad\": container with ID starting with 8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad not found: ID does not exist" containerID="8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad" Oct 30 06:24:19.077357 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.077317 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad"} err="failed to get container status \"8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad\": rpc error: code = NotFound desc = could not find container \"8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad\": container with ID starting with 8daaa64a6e2732b03b470dbd6bc41b3c196b16bcac466b32465e1c95e9d1ecad not found: ID does not exist" Oct 30 06:24:19.091070 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.091039 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw"] Oct 30 06:24:19.097158 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:19.097136 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-p85dw"] Oct 30 06:24:20.518525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:20.518494 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17588e96-befd-440e-aca9-e8582a61fcdf" path="/var/lib/kubelet/pods/17588e96-befd-440e-aca9-e8582a61fcdf/volumes" Oct 30 06:24:20.518962 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:20.518948 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" path="/var/lib/kubelet/pods/cd71ff23-7006-4545-aeb3-087a3310a0bb/volumes" Oct 30 06:24:22.150395 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:22.150355 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" podUID="c0bd7978-6f64-456c-b403-108050d494e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:26.475320 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:26.472021 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t"] Oct 30 06:24:26.475320 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:26.472276 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" containerID="cri-o://cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f" gracePeriod=30 Oct 30 06:24:27.992510 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:27.992480 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:24:28.015663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.015634 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bd7978-6f64-456c-b403-108050d494e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bd7978-6f64-456c-b403-108050d494e2-plugins-config-volume\") pod \"c0bd7978-6f64-456c-b403-108050d494e2\" (UID: \"c0bd7978-6f64-456c-b403-108050d494e2\") " Oct 30 06:24:28.015856 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.015684 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bd7978-6f64-456c-b403-108050d494e2-kube-api-access-675lj\" (UniqueName: \"kubernetes.io/projected/c0bd7978-6f64-456c-b403-108050d494e2-kube-api-access-675lj\") pod \"c0bd7978-6f64-456c-b403-108050d494e2\" (UID: \"c0bd7978-6f64-456c-b403-108050d494e2\") " Oct 30 06:24:28.016070 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.016048 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bd7978-6f64-456c-b403-108050d494e2-plugins-config-volume" pod "c0bd7978-6f64-456c-b403-108050d494e2" (UID: "c0bd7978-6f64-456c-b403-108050d494e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:28.017912 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.017891 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bd7978-6f64-456c-b403-108050d494e2-kube-api-access-675lj" pod "c0bd7978-6f64-456c-b403-108050d494e2" (UID: "c0bd7978-6f64-456c-b403-108050d494e2"). InnerVolumeSpecName "kube-api-access-675lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:28.062750 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.062711 5072 generic.go:356] "Generic (PLEG): container finished" podID="c0bd7978-6f64-456c-b403-108050d494e2" containerID="7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc" exitCode=0 Oct 30 06:24:28.062938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.062771 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" Oct 30 06:24:28.062938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.062765 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" event={"ID":"c0bd7978-6f64-456c-b403-108050d494e2","Type":"ContainerDied","Data":"7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc"} Oct 30 06:24:28.062938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.062913 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx" event={"ID":"c0bd7978-6f64-456c-b403-108050d494e2","Type":"ContainerDied","Data":"12a55344856943b6b46251b164c6929c73979082e3acedf5fe1a686793050610"} Oct 30 06:24:28.063113 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.062949 5072 scope.go:122] "RemoveContainer" containerID="7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc" Oct 30 06:24:28.076001 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.075978 5072 scope.go:122] "RemoveContainer" containerID="7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc" Oct 30 06:24:28.076293 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:28.076271 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc\": container with ID starting with 7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc not found: ID does not exist" containerID="7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc" Oct 30 06:24:28.076367 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.076304 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc"} err="failed to get container status \"7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc\": rpc error: code = NotFound desc = could not find container \"7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc\": container with ID starting with 7e447d8239f7da21b543a048067166be46265a419bf8e3906c721b71f71443bc not found: ID does not exist" Oct 30 06:24:28.113693 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.113661 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx"] Oct 30 06:24:28.117239 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.117204 5072 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0bd7978-6f64-456c-b403-108050d494e2-plugins-config-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:28.117239 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.117229 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-675lj\" (UniqueName: \"kubernetes.io/projected/c0bd7978-6f64-456c-b403-108050d494e2-kube-api-access-675lj\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:28.117774 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.117751 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2f1e4c-6598ff9b5f-dbghx"] Oct 30 06:24:28.526883 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:28.526836 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bd7978-6f64-456c-b403-108050d494e2" path="/var/lib/kubelet/pods/c0bd7978-6f64-456c-b403-108050d494e2/volumes" Oct 30 06:24:29.538501 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.533587 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb"] Oct 30 06:24:29.538501 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.533855 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" podUID="416250e1-133e-4f0d-98e9-5d5d6c0f8105" containerName="maas-api" containerID="cri-o://e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7" gracePeriod=30 Oct 30 06:24:29.801024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.800956 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 06:24:29.839036 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.839007 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls\") pod \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " Oct 30 06:24:29.839214 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.839102 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416250e1-133e-4f0d-98e9-5d5d6c0f8105-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416250e1-133e-4f0d-98e9-5d5d6c0f8105-openshift-service-ca\") pod \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " Oct 30 06:24:29.839214 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.839130 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416250e1-133e-4f0d-98e9-5d5d6c0f8105-kube-api-access-xmpx4\" (UniqueName: \"kubernetes.io/projected/416250e1-133e-4f0d-98e9-5d5d6c0f8105-kube-api-access-xmpx4\") pod \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " Oct 30 06:24:29.839214 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.839184 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-metrics-certs\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-metrics-certs\") pod \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\" (UID: \"416250e1-133e-4f0d-98e9-5d5d6c0f8105\") " Oct 30 06:24:29.839611 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.839536 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416250e1-133e-4f0d-98e9-5d5d6c0f8105-openshift-service-ca" pod "416250e1-133e-4f0d-98e9-5d5d6c0f8105" (UID: "416250e1-133e-4f0d-98e9-5d5d6c0f8105"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:29.842006 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.841967 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-metrics-certs" pod "416250e1-133e-4f0d-98e9-5d5d6c0f8105" (UID: "416250e1-133e-4f0d-98e9-5d5d6c0f8105"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:29.842006 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.841977 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416250e1-133e-4f0d-98e9-5d5d6c0f8105-kube-api-access-xmpx4" pod "416250e1-133e-4f0d-98e9-5d5d6c0f8105" (UID: "416250e1-133e-4f0d-98e9-5d5d6c0f8105"). InnerVolumeSpecName "kube-api-access-xmpx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:29.842187 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.842054 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls" pod "416250e1-133e-4f0d-98e9-5d5d6c0f8105" (UID: "416250e1-133e-4f0d-98e9-5d5d6c0f8105"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:29.940712 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.940681 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.940712 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.940709 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416250e1-133e-4f0d-98e9-5d5d6c0f8105-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.940712 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.940720 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmpx4\" (UniqueName: \"kubernetes.io/projected/416250e1-133e-4f0d-98e9-5d5d6c0f8105-kube-api-access-xmpx4\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.941024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:29.940730 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416250e1-133e-4f0d-98e9-5d5d6c0f8105-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:30.077320 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.077215 5072 generic.go:356] "Generic (PLEG): container finished" podID="416250e1-133e-4f0d-98e9-5d5d6c0f8105" containerID="e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7" exitCode=0 Oct 30 06:24:30.077320 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.077270 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" Oct 30 06:24:30.077320 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.077277 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" event={"ID":"416250e1-133e-4f0d-98e9-5d5d6c0f8105","Type":"ContainerDied","Data":"e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7"} Oct 30 06:24:30.077642 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.077334 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb" event={"ID":"416250e1-133e-4f0d-98e9-5d5d6c0f8105","Type":"ContainerDied","Data":"db24669af5824db4e381e401c366114cbf480bd0c9fb8b29759564974eb9c2e0"} Oct 30 06:24:30.077642 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.077355 5072 scope.go:122] "RemoveContainer" containerID="e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7" Oct 30 06:24:30.087417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.087394 5072 scope.go:122] "RemoveContainer" containerID="e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7" Oct 30 06:24:30.088797 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:30.088766 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7\": container with ID starting with e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7 not found: ID does not exist" containerID="e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7" Oct 30 06:24:30.088882 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.088807 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7"} err="failed to get container status \"e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7\": rpc error: code = NotFound desc = could not find container \"e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7\": container with ID starting with e1be1b8d32ecfe515378a31e34e1dd0d82091a609e91fcf33422321f4095b8e7 not found: ID does not exist" Oct 30 06:24:30.132612 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.132567 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb"] Oct 30 06:24:30.136444 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.136418 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-64123f-5bdbb5574-wrpgb"] Oct 30 06:24:30.519477 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:30.519428 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416250e1-133e-4f0d-98e9-5d5d6c0f8105" path="/var/lib/kubelet/pods/416250e1-133e-4f0d-98e9-5d5d6c0f8105/volumes" Oct 30 06:24:30.711414 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:30.711377 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:24:31.751293 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.751234 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 06:24:31.859061 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859025 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-certs\") pod \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " Oct 30 06:24:31.859283 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859107 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-envoy\") pod \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " Oct 30 06:24:31.859283 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859137 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-socket\") pod \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " Oct 30 06:24:31.859283 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859176 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-data\") pod \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " Oct 30 06:24:31.859283 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859205 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istiod-ca-cert\") pod \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " Oct 30 06:24:31.859538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859316 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-credential-socket\") pod \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " Oct 30 06:24:31.859538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859356 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-kube-api-access-bgxpt\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-kube-api-access-bgxpt\") pod \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " Oct 30 06:24:31.859538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859373 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-certs" pod "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" (UID: "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:31.859538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859388 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-token\") pod \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " Oct 30 06:24:31.859538 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859420 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-podinfo\") pod \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\" (UID: \"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2\") " Oct 30 06:24:31.859781 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859543 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-socket" pod "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" (UID: "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:31.859857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859835 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:31.859929 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859871 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:31.859929 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.859837 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istiod-ca-cert" pod "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" (UID: "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:31.860054 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.860034 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-data" pod "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" (UID: "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:31.860121 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.860069 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-credential-socket" pod "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" (UID: "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:31.861790 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.861758 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-token" pod "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" (UID: "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:31.861938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.861882 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-envoy" pod "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" (UID: "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:31.862228 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.862207 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-podinfo" pod "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" (UID: "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:31.862313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.862234 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-kube-api-access-bgxpt" pod "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" (UID: "b4e940ff-2f2d-4c6d-bc35-84d57448ddf2"). InnerVolumeSpecName "kube-api-access-bgxpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:31.961198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.961162 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:31.961198 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.961198 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgxpt\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-kube-api-access-bgxpt\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:31.961431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.961213 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:31.961431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.961228 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:31.961431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.961243 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:31.961431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.961255 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:31.961431 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:31.961269 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:32.091817 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.091733 5072 generic.go:356] "Generic (PLEG): container finished" podID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerID="cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f" exitCode=0 Oct 30 06:24:32.091817 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.091790 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" Oct 30 06:24:32.092013 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.091793 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" event={"ID":"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2","Type":"ContainerDied","Data":"cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f"} Oct 30 06:24:32.092013 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.091897 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t" event={"ID":"b4e940ff-2f2d-4c6d-bc35-84d57448ddf2","Type":"ContainerDied","Data":"89f321a3753ad713d86c6f86f188f034db02acce4105e1f7b5f7cbea9b5e44e8"} Oct 30 06:24:32.092013 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.091924 5072 scope.go:122] "RemoveContainer" containerID="cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f" Oct 30 06:24:32.106701 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.106660 5072 scope.go:122] "RemoveContainer" containerID="8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8" Oct 30 06:24:32.119378 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.119353 5072 scope.go:122] "RemoveContainer" containerID="cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f" Oct 30 06:24:32.119749 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:32.119719 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f\": container with ID starting with cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f not found: ID does not exist" containerID="cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f" Oct 30 06:24:32.119876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.119760 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f"} err="failed to get container status \"cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f\": rpc error: code = NotFound desc = could not find container \"cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f\": container with ID starting with cc173a64a43c81d992e4ddc8d83cedeb1f36d69ac536e08206ff3751dff7a29f not found: ID does not exist" Oct 30 06:24:32.119876 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.119783 5072 scope.go:122] "RemoveContainer" containerID="8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8" Oct 30 06:24:32.120104 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:32.120076 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8\": container with ID starting with 8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8 not found: ID does not exist" containerID="8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8" Oct 30 06:24:32.120200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.120117 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8"} err="failed to get container status \"8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8\": rpc error: code = NotFound desc = could not find container \"8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8\": container with ID starting with 8068d9fb390bbf4be4b9c9e0a5d5eb40632fc005dc4c007f23105765cc93b7a8 not found: ID does not exist" Oct 30 06:24:32.143894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.143859 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t"] Oct 30 06:24:32.147978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.147953 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-b7w7t"] Oct 30 06:24:32.521736 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:32.521705 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" path="/var/lib/kubelet/pods/b4e940ff-2f2d-4c6d-bc35-84d57448ddf2/volumes" Oct 30 06:24:33.659534 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.659487 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" containerID="cri-o://ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf" gracePeriod=45 Oct 30 06:24:33.927699 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.927672 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:24:33.981830 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.981793 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-kube-api-access-xvf48\" (UniqueName: \"kubernetes.io/projected/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-kube-api-access-xvf48\") pod \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " Oct 30 06:24:33.982006 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.981871 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-model-cache\") pod \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " Oct 30 06:24:33.982006 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.981978 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tls-certs\" (UniqueName: \"kubernetes.io/secret/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tls-certs\") pod \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " Oct 30 06:24:33.982132 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.982042 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-dshm\") pod \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " Oct 30 06:24:33.982132 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.982083 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-home\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-home\") pod \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " Oct 30 06:24:33.982227 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.982144 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tmp-dir\") pod \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\" (UID: \"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9\") " Oct 30 06:24:33.982915 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.982884 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tmp-dir" pod "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" (UID: "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:33.984761 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.984733 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-home" pod "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" (UID: "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:33.984954 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.984922 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-model-cache" pod "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" (UID: "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:33.986653 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.986619 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-dshm" pod "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" (UID: "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:33.998294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:33.995780 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tls-certs" pod "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" (UID: "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:34.007159 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.007094 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-kube-api-access-xvf48" pod "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" (UID: "51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9"). InnerVolumeSpecName "kube-api-access-xvf48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:34.083220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.083186 5072 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-dshm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.083220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.083218 5072 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-home\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.083426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.083247 5072 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tmp-dir\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.083426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.083259 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvf48\" (UniqueName: \"kubernetes.io/projected/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-kube-api-access-xvf48\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.083426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.083270 5072 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-model-cache\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.083426 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.083278 5072 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9-tls-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.107978 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.107946 5072 generic.go:356] "Generic (PLEG): container finished" podID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerID="ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf" exitCode=0 Oct 30 06:24:34.108146 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.108002 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" Oct 30 06:24:34.108146 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.108030 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" event={"ID":"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9","Type":"ContainerDied","Data":"ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf"} Oct 30 06:24:34.108146 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.108079 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c" event={"ID":"51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9","Type":"ContainerDied","Data":"dc9b8e2904528737211922d1dcc6083148da06e5c775ba612e77ad919fd74771"} Oct 30 06:24:34.108146 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.108098 5072 scope.go:122] "RemoveContainer" containerID="ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf" Oct 30 06:24:34.118105 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.118086 5072 scope.go:122] "RemoveContainer" containerID="ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf" Oct 30 06:24:34.118349 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:34.118329 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf\": container with ID starting with ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf not found: ID does not exist" containerID="ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf" Oct 30 06:24:34.118436 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.118357 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf"} err="failed to get container status \"ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf\": rpc error: code = NotFound desc = could not find container \"ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf\": container with ID starting with ea01783496eb889013e978f97b47aba2bd8778d68c7e418be5f56ec88a0d1daf not found: ID does not exist" Oct 30 06:24:34.158668 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.158562 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c"] Oct 30 06:24:34.161669 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.161607 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-2-w5-be78f2-kserve-585df9d8bf-wgs7c"] Oct 30 06:24:34.517488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:34.517390 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" path="/var/lib/kubelet/pods/51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9/volumes" Oct 30 06:24:37.606440 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:37.606405 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8"] Oct 30 06:24:37.606925 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:37.606682 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" podUID="96a7a0ba-240b-45ae-9df6-ef1d323cde72" containerName="maas-api" containerID="cri-o://bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f" gracePeriod=30 Oct 30 06:24:37.894140 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:37.894114 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 06:24:38.023801 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.023772 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a7a0ba-240b-45ae-9df6-ef1d323cde72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a7a0ba-240b-45ae-9df6-ef1d323cde72-openshift-service-ca\") pod \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " Oct 30 06:24:38.023988 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.023822 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a7a0ba-240b-45ae-9df6-ef1d323cde72-kube-api-access-h4klb\" (UniqueName: \"kubernetes.io/projected/96a7a0ba-240b-45ae-9df6-ef1d323cde72-kube-api-access-h4klb\") pod \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " Oct 30 06:24:38.023988 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.023937 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls\") pod \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " Oct 30 06:24:38.023988 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.023983 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-metrics-certs\") pod \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\" (UID: \"96a7a0ba-240b-45ae-9df6-ef1d323cde72\") " Oct 30 06:24:38.024247 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.024219 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a7a0ba-240b-45ae-9df6-ef1d323cde72-openshift-service-ca" pod "96a7a0ba-240b-45ae-9df6-ef1d323cde72" (UID: "96a7a0ba-240b-45ae-9df6-ef1d323cde72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:38.026243 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.026219 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a7a0ba-240b-45ae-9df6-ef1d323cde72-kube-api-access-h4klb" pod "96a7a0ba-240b-45ae-9df6-ef1d323cde72" (UID: "96a7a0ba-240b-45ae-9df6-ef1d323cde72"). InnerVolumeSpecName "kube-api-access-h4klb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:38.026336 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.026260 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls" pod "96a7a0ba-240b-45ae-9df6-ef1d323cde72" (UID: "96a7a0ba-240b-45ae-9df6-ef1d323cde72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:38.026336 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.026325 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-metrics-certs" pod "96a7a0ba-240b-45ae-9df6-ef1d323cde72" (UID: "96a7a0ba-240b-45ae-9df6-ef1d323cde72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:38.125666 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.125591 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:38.125666 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.125617 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96a7a0ba-240b-45ae-9df6-ef1d323cde72-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:38.125666 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.125627 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a7a0ba-240b-45ae-9df6-ef1d323cde72-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:38.125666 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.125636 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4klb\" (UniqueName: \"kubernetes.io/projected/96a7a0ba-240b-45ae-9df6-ef1d323cde72-kube-api-access-h4klb\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:38.133169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.133133 5072 generic.go:356] "Generic (PLEG): container finished" podID="96a7a0ba-240b-45ae-9df6-ef1d323cde72" containerID="bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f" exitCode=0 Oct 30 06:24:38.133310 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.133180 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" Oct 30 06:24:38.133310 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.133212 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" event={"ID":"96a7a0ba-240b-45ae-9df6-ef1d323cde72","Type":"ContainerDied","Data":"bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f"} Oct 30 06:24:38.133310 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.133269 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8" event={"ID":"96a7a0ba-240b-45ae-9df6-ef1d323cde72","Type":"ContainerDied","Data":"586c9a69766f1785804c39322e105bf5c3ccbd24cdf30fd7bfbd797d732229fc"} Oct 30 06:24:38.133310 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.133293 5072 scope.go:122] "RemoveContainer" containerID="bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f" Oct 30 06:24:38.145729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.145195 5072 scope.go:122] "RemoveContainer" containerID="bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f" Oct 30 06:24:38.145841 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:38.145810 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f\": container with ID starting with bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f not found: ID does not exist" containerID="bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f" Oct 30 06:24:38.145955 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.145848 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f"} err="failed to get container status \"bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f\": rpc error: code = NotFound desc = could not find container \"bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f\": container with ID starting with bd77be308cec521d4816b59436ba10e8ed276d457fe84ddf655f2f40ab42dd5f not found: ID does not exist" Oct 30 06:24:38.214825 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.214794 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8"] Oct 30 06:24:38.239596 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.239560 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-be78f2-6d6847bf4c-x28l8"] Oct 30 06:24:38.518976 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:38.518886 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a7a0ba-240b-45ae-9df6-ef1d323cde72" path="/var/lib/kubelet/pods/96a7a0ba-240b-45ae-9df6-ef1d323cde72/volumes" Oct 30 06:24:51.218784 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.218685 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b97b5857-2ppc5"] Oct 30 06:24:51.219365 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.218984 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7b97b5857-2ppc5" podUID="e9a793bc-54e2-4bb3-aaac-ec7a7d94467c" containerName="manager" containerID="cri-o://a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c" gracePeriod=30 Oct 30 06:24:51.490071 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.490041 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 06:24:51.663123 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.662931 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-webhook-cert\") pod \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " Oct 30 06:24:51.663123 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.663004 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-metrics-certs\") pod \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " Oct 30 06:24:51.663123 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.663081 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-kube-api-access-cvthp\" (UniqueName: \"kubernetes.io/projected/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-kube-api-access-cvthp\") pod \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\" (UID: \"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c\") " Oct 30 06:24:51.666132 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.666100 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-kube-api-access-cvthp" pod "e9a793bc-54e2-4bb3-aaac-ec7a7d94467c" (UID: "e9a793bc-54e2-4bb3-aaac-ec7a7d94467c"). InnerVolumeSpecName "kube-api-access-cvthp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:51.667063 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.667036 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-webhook-cert" pod "e9a793bc-54e2-4bb3-aaac-ec7a7d94467c" (UID: "e9a793bc-54e2-4bb3-aaac-ec7a7d94467c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:51.668798 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.668771 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-metrics-certs" pod "e9a793bc-54e2-4bb3-aaac-ec7a7d94467c" (UID: "e9a793bc-54e2-4bb3-aaac-ec7a7d94467c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:51.765547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.765425 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvthp\" (UniqueName: \"kubernetes.io/projected/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-kube-api-access-cvthp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.765547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.765489 5072 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-webhook-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.765547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:51.765503 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:24:52.220701 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.220660 5072 generic.go:356] "Generic (PLEG): container finished" podID="e9a793bc-54e2-4bb3-aaac-ec7a7d94467c" containerID="a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c" exitCode=0 Oct 30 06:24:52.221203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.220720 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-2ppc5" Oct 30 06:24:52.221203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.220750 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-2ppc5" event={"ID":"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c","Type":"ContainerDied","Data":"a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c"} Oct 30 06:24:52.221203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.220821 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-2ppc5" event={"ID":"e9a793bc-54e2-4bb3-aaac-ec7a7d94467c","Type":"ContainerDied","Data":"4114062471c03b9e08859adb30fc6e812fa29987bb5d4dec31f8cf3828b318e4"} Oct 30 06:24:52.221203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.220855 5072 scope.go:122] "RemoveContainer" containerID="a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c" Oct 30 06:24:52.237416 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.237399 5072 scope.go:122] "RemoveContainer" containerID="a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c" Oct 30 06:24:52.237745 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:24:52.237724 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c\": container with ID starting with a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c not found: ID does not exist" containerID="a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c" Oct 30 06:24:52.237857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.237757 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c"} err="failed to get container status \"a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c\": rpc error: code = NotFound desc = could not find container \"a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c\": container with ID starting with a592b5fcedaf64c2bc457373c4c9f017a3a6fdc684f8f9e4d19fb635f3bbd72c not found: ID does not exist" Oct 30 06:24:52.272281 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.272242 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b97b5857-2ppc5"] Oct 30 06:24:52.284656 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.284629 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b97b5857-2ppc5"] Oct 30 06:24:52.519665 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.519585 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a793bc-54e2-4bb3-aaac-ec7a7d94467c" path="/var/lib/kubelet/pods/e9a793bc-54e2-4bb3-aaac-ec7a7d94467c/volumes" Oct 30 06:24:52.995575 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.994996 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b97b5857-6jzph"] Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995617 5072 cpu_manager.go:414] "containerMap: removing container" podUID="12551d54-2727-4931-8ac4-4023d7613f83" containerName="istio-proxy" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995636 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12551d54-2727-4931-8ac4-4023d7613f83" containerName="istio-proxy" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995650 5072 cpu_manager.go:414] "containerMap: removing container" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995657 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995670 5072 cpu_manager.go:414] "containerMap: removing container" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995677 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995686 5072 cpu_manager.go:414] "containerMap: removing container" podUID="416250e1-133e-4f0d-98e9-5d5d6c0f8105" containerName="maas-api" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995693 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416250e1-133e-4f0d-98e9-5d5d6c0f8105" containerName="maas-api" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995701 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b7183d70-926a-4483-a268-3dc50b09571d" containerName="maas-api" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995709 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7183d70-926a-4483-a268-3dc50b09571d" containerName="maas-api" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995718 5072 cpu_manager.go:414] "containerMap: removing container" podUID="74387953-1299-4d3b-8e25-995bd29ce315" containerName="payload-pre-processing" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995724 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74387953-1299-4d3b-8e25-995bd29ce315" containerName="payload-pre-processing" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995738 5072 cpu_manager.go:414] "containerMap: removing container" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995744 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995757 5072 cpu_manager.go:414] "containerMap: removing container" podUID="0359cfa7-f684-476e-a9e6-a2defd1feb2b" containerName="maas-api" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995764 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0359cfa7-f684-476e-a9e6-a2defd1feb2b" containerName="maas-api" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995773 5072 cpu_manager.go:414] "containerMap: removing container" podUID="c0bd7978-6f64-456c-b403-108050d494e2" containerName="payload-pre-processing" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995778 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bd7978-6f64-456c-b403-108050d494e2" containerName="payload-pre-processing" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995787 5072 cpu_manager.go:414] "containerMap: removing container" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" Oct 30 06:24:52.995788 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995794 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995805 5072 cpu_manager.go:414] "containerMap: removing container" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995811 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995821 5072 cpu_manager.go:414] "containerMap: removing container" podUID="76a5be12-b842-446d-b841-3af5bf5dde44" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995829 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a5be12-b842-446d-b841-3af5bf5dde44" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995844 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995852 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995862 5072 cpu_manager.go:414] "containerMap: removing container" podUID="96a7a0ba-240b-45ae-9df6-ef1d323cde72" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995868 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a7a0ba-240b-45ae-9df6-ef1d323cde72" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995878 5072 cpu_manager.go:414] "containerMap: removing container" podUID="9a235824-ceb8-4424-9404-64c98bd44b39" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995885 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a235824-ceb8-4424-9404-64c98bd44b39" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995905 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995912 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995924 5072 cpu_manager.go:414] "containerMap: removing container" podUID="5ea933c4-0f19-489d-a553-98d2da67114a" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995930 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea933c4-0f19-489d-a553-98d2da67114a" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995942 5072 cpu_manager.go:414] "containerMap: removing container" podUID="17588e96-befd-440e-aca9-e8582a61fcdf" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995948 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17588e96-befd-440e-aca9-e8582a61fcdf" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995959 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995965 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995974 5072 cpu_manager.go:414] "containerMap: removing container" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995980 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995989 5072 cpu_manager.go:414] "containerMap: removing container" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.995997 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996007 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996014 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996027 5072 cpu_manager.go:414] "containerMap: removing container" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996034 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996050 5072 cpu_manager.go:414] "containerMap: removing container" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996057 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996069 5072 cpu_manager.go:414] "containerMap: removing container" podUID="d74d09be-6408-4b4f-bdbf-f36c2953566f" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996076 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74d09be-6408-4b4f-bdbf-f36c2953566f" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996085 5072 cpu_manager.go:414] "containerMap: removing container" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996092 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996114 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996122 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996138 5072 cpu_manager.go:414] "containerMap: removing container" podUID="b04f54d9-44d0-4fae-af50-c0c6e20df84c" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996145 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04f54d9-44d0-4fae-af50-c0c6e20df84c" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996170 5072 cpu_manager.go:414] "containerMap: removing container" podUID="28026c62-1c09-45d4-9d3e-810c67b7117c" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996178 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28026c62-1c09-45d4-9d3e-810c67b7117c" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996188 5072 cpu_manager.go:414] "containerMap: removing container" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996195 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996207 5072 cpu_manager.go:414] "containerMap: removing container" podUID="e9a793bc-54e2-4bb3-aaac-ec7a7d94467c" containerName="manager" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996214 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a793bc-54e2-4bb3-aaac-ec7a7d94467c" containerName="manager" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996232 5072 cpu_manager.go:414] "containerMap: removing container" podUID="2b0d06fe-f98e-4c1b-8669-aa8ee23164f5" containerName="payload-processing" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996240 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0d06fe-f98e-4c1b-8669-aa8ee23164f5" containerName="payload-processing" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996257 5072 cpu_manager.go:414] "containerMap: removing container" podUID="af021843-9c8f-4ce6-8f08-858104de58dc" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996265 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af021843-9c8f-4ce6-8f08-858104de58dc" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996276 5072 cpu_manager.go:414] "containerMap: removing container" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996283 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996293 5072 cpu_manager.go:414] "containerMap: removing container" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996300 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996313 5072 cpu_manager.go:414] "containerMap: removing container" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996320 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996337 5072 cpu_manager.go:414] "containerMap: removing container" podUID="788e07b5-328b-4d53-bb7d-c82b74d604d6" containerName="payload-processing" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996345 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788e07b5-328b-4d53-bb7d-c82b74d604d6" containerName="payload-processing" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996487 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="12551d54-2727-4931-8ac4-4023d7613f83" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996505 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30d89ef-de1f-493d-b956-70ffc9d4943c" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996513 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996521 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="416250e1-133e-4f0d-98e9-5d5d6c0f8105" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996531 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7183d70-926a-4483-a268-3dc50b09571d" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996543 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="74387953-1299-4d3b-8e25-995bd29ce315" containerName="payload-pre-processing" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996553 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b8e9c3-0956-46e3-bcb7-02225a62c460" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996564 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0359cfa7-f684-476e-a9e6-a2defd1feb2b" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996572 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bd7978-6f64-456c-b403-108050d494e2" containerName="payload-pre-processing" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996581 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996590 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996599 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a5be12-b842-446d-b841-3af5bf5dde44" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996608 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996617 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a7a0ba-240b-45ae-9df6-ef1d323cde72" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996626 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a235824-ceb8-4424-9404-64c98bd44b39" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996638 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996647 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea933c4-0f19-489d-a553-98d2da67114a" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996656 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="17588e96-befd-440e-aca9-e8582a61fcdf" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996664 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7452308-ea76-448e-a885-300436798e92" containerName="main" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996673 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996682 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd71ff23-7006-4545-aeb3-087a3310a0bb" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996690 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b959f379-69f3-4be5-ac96-8fd4bc2211be" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996701 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996711 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="51afb6ee-e6d5-4f7f-a6c8-0b0c8bd9fac9" containerName="main" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996721 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996731 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74d09be-6408-4b4f-bdbf-f36c2953566f" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996739 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e940ff-2f2d-4c6d-bc35-84d57448ddf2" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996750 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04f54d9-44d0-4fae-af50-c0c6e20df84c" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996760 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="28026c62-1c09-45d4-9d3e-810c67b7117c" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996769 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54f773d-5f0c-4bde-a25f-5385934a95e1" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996778 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a793bc-54e2-4bb3-aaac-ec7a7d94467c" containerName="manager" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996787 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0d06fe-f98e-4c1b-8669-aa8ee23164f5" containerName="payload-processing" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996798 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="af021843-9c8f-4ce6-8f08-858104de58dc" containerName="maas-api" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996805 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf8ac2d-724c-4309-aaba-de0952fc9802" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996814 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb930a6-978c-413f-b1fb-6b7ab90d5fc0" containerName="istio-proxy" Oct 30 06:24:52.996985 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:52.996822 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="788e07b5-328b-4d53-bb7d-c82b74d604d6" containerName="payload-processing" Oct 30 06:24:53.001932 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.001910 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.033660 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.033639 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6ttmw\"" Oct 30 06:24:53.033793 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.033770 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 06:24:53.033872 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.033853 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 06:24:53.035600 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.035580 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b97b5857-6jzph"] Oct 30 06:24:53.114014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.113984 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 06:24:53.179535 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.179503 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstmv\" (UniqueName: \"kubernetes.io/projected/052435eb-4b43-43a8-80ba-800a3c7a210d-kube-api-access-nstmv\") pod \"maas-controller-7b97b5857-6jzph\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.179681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.179573 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-webhook-cert\") pod \"maas-controller-7b97b5857-6jzph\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.179681 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.179631 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-metrics-certs\") pod \"maas-controller-7b97b5857-6jzph\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.280910 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.280832 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstmv\" (UniqueName: \"kubernetes.io/projected/052435eb-4b43-43a8-80ba-800a3c7a210d-kube-api-access-nstmv\") pod \"maas-controller-7b97b5857-6jzph\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.280910 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.280882 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-webhook-cert\") pod \"maas-controller-7b97b5857-6jzph\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.281419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.281139 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-metrics-certs\") pod \"maas-controller-7b97b5857-6jzph\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.283583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.283552 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-webhook-cert\") pod \"maas-controller-7b97b5857-6jzph\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.283583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.283575 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-metrics-certs\") pod \"maas-controller-7b97b5857-6jzph\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.304638 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.304611 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstmv\" (UniqueName: \"kubernetes.io/projected/052435eb-4b43-43a8-80ba-800a3c7a210d-kube-api-access-nstmv\") pod \"maas-controller-7b97b5857-6jzph\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.315273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.315257 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:53.613934 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.613907 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b97b5857-6jzph"] Oct 30 06:24:53.615141 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:24:53.615110 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052435eb_4b43_43a8_80ba_800a3c7a210d.slice/crio-521c5862cf50d85cba70ffca00c8bab6974559392103ba024e890743feaca0a3 WatchSource:0}: Error finding container 521c5862cf50d85cba70ffca00c8bab6974559392103ba024e890743feaca0a3: Status 404 returned error can't find the container with id 521c5862cf50d85cba70ffca00c8bab6974559392103ba024e890743feaca0a3 Oct 30 06:24:53.616497 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:53.616475 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:24:54.233152 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:54.233111 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-6jzph" event={"ID":"052435eb-4b43-43a8-80ba-800a3c7a210d","Type":"ContainerStarted","Data":"dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a"} Oct 30 06:24:54.233308 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:54.233178 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-6jzph" event={"ID":"052435eb-4b43-43a8-80ba-800a3c7a210d","Type":"ContainerStarted","Data":"521c5862cf50d85cba70ffca00c8bab6974559392103ba024e890743feaca0a3"} Oct 30 06:24:54.233308 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:54.233209 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:24:54.275608 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:24:54.275557 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b97b5857-6jzph" podStartSLOduration=1.761972923 podStartE2EDuration="2.275544856s" totalImagesPullingTime="513.571933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:54.275023367 +0000 UTC m=+3253.878712624" watchObservedRunningTime="2026-10-30 06:24:54.275544856 +0000 UTC m=+3253.879234133" Oct 30 06:25:00.709879 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:25:00.709834 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:25:05.240160 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:05.240134 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:25:25.574847 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.574806 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv"] Oct 30 06:25:25.575341 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.575244 5072 cpu_manager.go:414] "containerMap: removing container" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" Oct 30 06:25:25.575341 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.575254 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1a82a3-3f78-4053-a9e6-3ca484c81f3a" containerName="istio-proxy" Oct 30 06:25:25.575341 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.575334 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b49a18-306a-4cbe-8fe3-1295c43ca4f3" containerName="istio-proxy" Oct 30 06:25:25.575487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.575346 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d97d10-ff5d-499a-a994-2721e2e3c8fa" containerName="istio-proxy" Oct 30 06:25:25.579028 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.579007 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.607911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.607880 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv"] Oct 30 06:25:25.700735 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.700696 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.700735 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.700749 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.700959 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.700818 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.700959 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.700867 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.700959 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.700924 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.701069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.700969 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.701069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.701003 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.701069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.701028 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.701175 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.701116 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7h5v\" (UniqueName: \"kubernetes.io/projected/a188fbf5-5b98-49b8-a241-11e0e437ee1a-kube-api-access-j7h5v\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802201 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802161 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802210 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802249 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802268 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802286 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802315 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802351 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802389 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802385 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802892 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802422 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7h5v\" (UniqueName: \"kubernetes.io/projected/a188fbf5-5b98-49b8-a241-11e0e437ee1a-kube-api-access-j7h5v\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802892 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802663 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802892 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802706 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802892 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802772 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.802892 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.802818 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.803170 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.803152 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.804890 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.804869 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.805097 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.805078 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.825794 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.825730 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a188fbf5-5b98-49b8-a241-11e0e437ee1a-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.826061 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.826043 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7h5v\" (UniqueName: \"kubernetes.io/projected/a188fbf5-5b98-49b8-a241-11e0e437ee1a-kube-api-access-j7h5v\") pod \"maas-default-gateway-openshift-default-84587756dc-plrsv\" (UID: \"a188fbf5-5b98-49b8-a241-11e0e437ee1a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:25.892826 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:25.892773 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:26.080183 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.080098 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b97b5857-6jzph"] Oct 30 06:25:26.080345 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.080308 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7b97b5857-6jzph" podUID="052435eb-4b43-43a8-80ba-800a3c7a210d" containerName="manager" containerID="cri-o://dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a" gracePeriod=30 Oct 30 06:25:26.207598 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.207568 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv"] Oct 30 06:25:26.212248 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:25:26.212220 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda188fbf5_5b98_49b8_a241_11e0e437ee1a.slice/crio-ae7032aeebda8b78e2775a4fb5859afa46066874180927c4c9402fe7415a5061 WatchSource:0}: Error finding container ae7032aeebda8b78e2775a4fb5859afa46066874180927c4c9402fe7415a5061: Status 404 returned error can't find the container with id ae7032aeebda8b78e2775a4fb5859afa46066874180927c4c9402fe7415a5061 Oct 30 06:25:26.214037 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.213999 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:25:26.214156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.214087 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:25:26.214156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.214125 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:25:26.333123 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.333040 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:25:26.398012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.397715 5072 generic.go:356] "Generic (PLEG): container finished" podID="052435eb-4b43-43a8-80ba-800a3c7a210d" containerID="dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a" exitCode=0 Oct 30 06:25:26.398012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.397878 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-6jzph" Oct 30 06:25:26.398268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.398025 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-6jzph" event={"ID":"052435eb-4b43-43a8-80ba-800a3c7a210d","Type":"ContainerDied","Data":"dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a"} Oct 30 06:25:26.398268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.398085 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-6jzph" event={"ID":"052435eb-4b43-43a8-80ba-800a3c7a210d","Type":"ContainerDied","Data":"521c5862cf50d85cba70ffca00c8bab6974559392103ba024e890743feaca0a3"} Oct 30 06:25:26.398268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.398115 5072 scope.go:122] "RemoveContainer" containerID="dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a" Oct 30 06:25:26.400896 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.400650 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" event={"ID":"a188fbf5-5b98-49b8-a241-11e0e437ee1a","Type":"ContainerStarted","Data":"1e7a6938ed06bbdae9ea4404ced25fc6595f7a17fb6e330e53158a2733c23e4d"} Oct 30 06:25:26.400896 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.400726 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" event={"ID":"a188fbf5-5b98-49b8-a241-11e0e437ee1a","Type":"ContainerStarted","Data":"ae7032aeebda8b78e2775a4fb5859afa46066874180927c4c9402fe7415a5061"} Oct 30 06:25:26.409163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.408439 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052435eb-4b43-43a8-80ba-800a3c7a210d-kube-api-access-nstmv\" (UniqueName: \"kubernetes.io/projected/052435eb-4b43-43a8-80ba-800a3c7a210d-kube-api-access-nstmv\") pod \"052435eb-4b43-43a8-80ba-800a3c7a210d\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " Oct 30 06:25:26.409163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.408552 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-metrics-certs\") pod \"052435eb-4b43-43a8-80ba-800a3c7a210d\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " Oct 30 06:25:26.409163 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.408621 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-webhook-cert\") pod \"052435eb-4b43-43a8-80ba-800a3c7a210d\" (UID: \"052435eb-4b43-43a8-80ba-800a3c7a210d\") " Oct 30 06:25:26.414358 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.414326 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-metrics-certs" pod "052435eb-4b43-43a8-80ba-800a3c7a210d" (UID: "052435eb-4b43-43a8-80ba-800a3c7a210d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:26.414358 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.414354 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052435eb-4b43-43a8-80ba-800a3c7a210d-kube-api-access-nstmv" pod "052435eb-4b43-43a8-80ba-800a3c7a210d" (UID: "052435eb-4b43-43a8-80ba-800a3c7a210d"). InnerVolumeSpecName "kube-api-access-nstmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:26.416117 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.416082 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-webhook-cert" pod "052435eb-4b43-43a8-80ba-800a3c7a210d" (UID: "052435eb-4b43-43a8-80ba-800a3c7a210d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:26.418058 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.418030 5072 scope.go:122] "RemoveContainer" containerID="dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a" Oct 30 06:25:26.418442 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:25:26.418416 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a\": container with ID starting with dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a not found: ID does not exist" containerID="dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a" Oct 30 06:25:26.418571 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.418473 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a"} err="failed to get container status \"dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a\": rpc error: code = NotFound desc = could not find container \"dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a\": container with ID starting with dad3a09d672e03a914271791655ecaef2d0777145b0cd0afd301cc42e9e2e90a not found: ID does not exist" Oct 30 06:25:26.460910 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.460859 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" podStartSLOduration=1.460842812 podStartE2EDuration="1.460842812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:26.459090883 +0000 UTC m=+3286.062780139" watchObservedRunningTime="2026-10-30 06:25:26.460842812 +0000 UTC m=+3286.064532070" Oct 30 06:25:26.510226 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.510180 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:25:26.510226 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.510209 5072 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/052435eb-4b43-43a8-80ba-800a3c7a210d-webhook-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:25:26.510226 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.510222 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nstmv\" (UniqueName: \"kubernetes.io/projected/052435eb-4b43-43a8-80ba-800a3c7a210d-kube-api-access-nstmv\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:25:26.740939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.740885 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b97b5857-6jzph"] Oct 30 06:25:26.751768 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.751743 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b97b5857-6jzph"] Oct 30 06:25:26.893718 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:26.893681 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:27.895105 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.895071 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-plrsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 06:25:27.895541 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.895125 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" podUID="a188fbf5-5b98-49b8-a241-11e0e437ee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 06:25:27.946200 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.946170 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b97b5857-55xsq"] Oct 30 06:25:27.946720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.946701 5072 cpu_manager.go:414] "containerMap: removing container" podUID="052435eb-4b43-43a8-80ba-800a3c7a210d" containerName="manager" Oct 30 06:25:27.946821 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.946722 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052435eb-4b43-43a8-80ba-800a3c7a210d" containerName="manager" Oct 30 06:25:27.946901 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.946828 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="052435eb-4b43-43a8-80ba-800a3c7a210d" containerName="manager" Oct 30 06:25:27.951279 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.951249 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:27.978157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.978106 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 06:25:27.978157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.978131 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 06:25:27.978157 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.978131 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6ttmw\"" Oct 30 06:25:27.978434 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.978316 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 06:25:27.980721 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:27.980694 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b97b5857-55xsq"] Oct 30 06:25:28.039784 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.039742 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpg6\" (UniqueName: \"kubernetes.io/projected/de98d103-370c-4da2-8244-500a482f16a1-kube-api-access-wfpg6\") pod \"maas-controller-7b97b5857-55xsq\" (UID: \"de98d103-370c-4da2-8244-500a482f16a1\") " pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.040000 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.039793 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de98d103-370c-4da2-8244-500a482f16a1-metrics-certs\") pod \"maas-controller-7b97b5857-55xsq\" (UID: \"de98d103-370c-4da2-8244-500a482f16a1\") " pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.040000 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.039923 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de98d103-370c-4da2-8244-500a482f16a1-webhook-cert\") pod \"maas-controller-7b97b5857-55xsq\" (UID: \"de98d103-370c-4da2-8244-500a482f16a1\") " pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.140769 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.140732 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpg6\" (UniqueName: \"kubernetes.io/projected/de98d103-370c-4da2-8244-500a482f16a1-kube-api-access-wfpg6\") pod \"maas-controller-7b97b5857-55xsq\" (UID: \"de98d103-370c-4da2-8244-500a482f16a1\") " pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.140769 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.140776 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de98d103-370c-4da2-8244-500a482f16a1-metrics-certs\") pod \"maas-controller-7b97b5857-55xsq\" (UID: \"de98d103-370c-4da2-8244-500a482f16a1\") " pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.141013 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.140839 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de98d103-370c-4da2-8244-500a482f16a1-webhook-cert\") pod \"maas-controller-7b97b5857-55xsq\" (UID: \"de98d103-370c-4da2-8244-500a482f16a1\") " pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.143594 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.143562 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de98d103-370c-4da2-8244-500a482f16a1-webhook-cert\") pod \"maas-controller-7b97b5857-55xsq\" (UID: \"de98d103-370c-4da2-8244-500a482f16a1\") " pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.143594 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.143587 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de98d103-370c-4da2-8244-500a482f16a1-metrics-certs\") pod \"maas-controller-7b97b5857-55xsq\" (UID: \"de98d103-370c-4da2-8244-500a482f16a1\") " pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.164155 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.164095 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpg6\" (UniqueName: \"kubernetes.io/projected/de98d103-370c-4da2-8244-500a482f16a1-kube-api-access-wfpg6\") pod \"maas-controller-7b97b5857-55xsq\" (UID: \"de98d103-370c-4da2-8244-500a482f16a1\") " pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.264080 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.264045 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:28.541188 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.541118 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052435eb-4b43-43a8-80ba-800a3c7a210d" path="/var/lib/kubelet/pods/052435eb-4b43-43a8-80ba-800a3c7a210d/volumes" Oct 30 06:25:28.580030 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.580001 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b97b5857-55xsq"] Oct 30 06:25:28.580622 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:25:28.580597 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde98d103_370c_4da2_8244_500a482f16a1.slice/crio-0c8db0ddab2227ef1726e7bdbab9db33df49a737eb52670660ade3be3682f5b7 WatchSource:0}: Error finding container 0c8db0ddab2227ef1726e7bdbab9db33df49a737eb52670660ade3be3682f5b7: Status 404 returned error can't find the container with id 0c8db0ddab2227ef1726e7bdbab9db33df49a737eb52670660ade3be3682f5b7 Oct 30 06:25:28.893372 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.893334 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-plrsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 06:25:28.893593 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:28.893390 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" podUID="a188fbf5-5b98-49b8-a241-11e0e437ee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 06:25:29.442623 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:29.442591 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-55xsq" event={"ID":"de98d103-370c-4da2-8244-500a482f16a1","Type":"ContainerStarted","Data":"44ae85bc5ce0022d70704f33355f5a082da9f6cc22bc914935d3b5fa95088127"} Oct 30 06:25:29.443118 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:29.442642 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b97b5857-55xsq" event={"ID":"de98d103-370c-4da2-8244-500a482f16a1","Type":"ContainerStarted","Data":"0c8db0ddab2227ef1726e7bdbab9db33df49a737eb52670660ade3be3682f5b7"} Oct 30 06:25:29.443118 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:29.442770 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:29.493474 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:29.493407 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b97b5857-55xsq" podStartSLOduration=2.040641635 podStartE2EDuration="2.493392742s" totalImagesPullingTime="452.751107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:29.485333202 +0000 UTC m=+3289.089022459" watchObservedRunningTime="2026-10-30 06:25:29.493392742 +0000 UTC m=+3289.097082000" Oct 30 06:25:29.893135 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:29.893099 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-plrsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 06:25:29.893355 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:29.893160 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" podUID="a188fbf5-5b98-49b8-a241-11e0e437ee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 06:25:30.734003 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:25:30.733816 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:25:30.893567 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:30.893531 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-plrsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 06:25:30.893811 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:30.893601 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" podUID="a188fbf5-5b98-49b8-a241-11e0e437ee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 06:25:31.893556 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:31.893508 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-plrsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 06:25:31.894132 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:31.893590 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" podUID="a188fbf5-5b98-49b8-a241-11e0e437ee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 06:25:32.893471 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:32.893427 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-plrsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 06:25:32.893645 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:32.893500 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" podUID="a188fbf5-5b98-49b8-a241-11e0e437ee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 06:25:34.460945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:34.460910 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:34.461429 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:34.461406 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:34.462199 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:34.462180 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-plrsv" Oct 30 06:25:40.454082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:40.454052 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b97b5857-55xsq" Oct 30 06:25:48.626383 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:48.626344 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:25:48.629107 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:25:48.629078 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:26:00.709694 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:26:00.709651 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:26:25.736377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.736336 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls"] Oct 30 06:26:25.740570 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.740547 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.777777 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.777397 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls"] Oct 30 06:26:25.779558 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.779533 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn"] Oct 30 06:26:25.783770 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.783745 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.805872 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.805841 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn"] Oct 30 06:26:25.835028 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835000 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.835178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835033 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.835178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835052 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.835178 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835075 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldnp7\" (UniqueName: \"kubernetes.io/projected/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-kube-api-access-ldnp7\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.835294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835187 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.835294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835220 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.835294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835242 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.835294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835260 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.835425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835347 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.835425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835381 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.835425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835408 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.835549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835443 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.835549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835508 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.835549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835532 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.835652 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835550 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.835652 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835567 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.835652 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835592 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.835652 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.835624 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw25z\" (UniqueName: \"kubernetes.io/projected/9862ec03-2fe1-42dc-8f6b-305348cc8bae-kube-api-access-cw25z\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.936737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.936704 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.936945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.936745 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.936945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.936765 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.936945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.936882 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.936945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.936928 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.937189 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.936969 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937189 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937012 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937189 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937043 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937189 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937156 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937189 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937160 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.937453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937205 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937209 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.937453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937265 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937303 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937332 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.937453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937353 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.937453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937377 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw25z\" (UniqueName: \"kubernetes.io/projected/9862ec03-2fe1-42dc-8f6b-305348cc8bae-kube-api-access-cw25z\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937389 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937499 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.937897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937537 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.937897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937563 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937607 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldnp7\" (UniqueName: \"kubernetes.io/projected/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-kube-api-access-ldnp7\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.937897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937657 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.937897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.937793 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.938220 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.938169 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.938444 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.938416 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.938655 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.938499 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.938783 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.938527 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.940174 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.940147 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.940288 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.940172 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.940799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.940776 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.941108 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.941085 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.961928 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.961898 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.962761 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.962742 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9862ec03-2fe1-42dc-8f6b-305348cc8bae-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:25.963183 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.963163 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldnp7\" (UniqueName: \"kubernetes.io/projected/92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3-kube-api-access-ldnp7\") pod \"maas-default-gateway-openshift-default-84587756dc-q5bls\" (UID: \"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:25.963388 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:25.963368 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw25z\" (UniqueName: \"kubernetes.io/projected/9862ec03-2fe1-42dc-8f6b-305348cc8bae-kube-api-access-cw25z\") pod \"maas-default-gateway-openshift-default-84587756dc-7lvmn\" (UID: \"9862ec03-2fe1-42dc-8f6b-305348cc8bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:26.056122 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.056036 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:26.097453 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.097421 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:26.373919 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.373891 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls"] Oct 30 06:26:26.374495 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:26:26.374438 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d3ce69_bc5b_4fc9_b918_1f390e1ed2e3.slice/crio-9c68ab09215cda5facc90318f685c1589a6f05e8ed559279bc5ca5f599087a0a WatchSource:0}: Error finding container 9c68ab09215cda5facc90318f685c1589a6f05e8ed559279bc5ca5f599087a0a: Status 404 returned error can't find the container with id 9c68ab09215cda5facc90318f685c1589a6f05e8ed559279bc5ca5f599087a0a Oct 30 06:26:26.376637 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.376580 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:26:26.376714 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.376670 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:26:26.376714 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.376697 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:26:26.417353 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.417324 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn"] Oct 30 06:26:26.418012 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:26:26.417991 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9862ec03_2fe1_42dc_8f6b_305348cc8bae.slice/crio-e15b13ac69d83d52b69fe62aeac58b6da360ba4c463124154392b930abf4dd95 WatchSource:0}: Error finding container e15b13ac69d83d52b69fe62aeac58b6da360ba4c463124154392b930abf4dd95: Status 404 returned error can't find the container with id e15b13ac69d83d52b69fe62aeac58b6da360ba4c463124154392b930abf4dd95 Oct 30 06:26:26.420384 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.420356 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:26:26.420480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.420417 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:26:26.420480 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.420444 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:26:26.750145 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.750039 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" event={"ID":"9862ec03-2fe1-42dc-8f6b-305348cc8bae","Type":"ContainerStarted","Data":"2d96d9f919fc6287242239f946bf06722632d37242f7813b74080fa7e36f328a"} Oct 30 06:26:26.750145 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.750133 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" event={"ID":"9862ec03-2fe1-42dc-8f6b-305348cc8bae","Type":"ContainerStarted","Data":"e15b13ac69d83d52b69fe62aeac58b6da360ba4c463124154392b930abf4dd95"} Oct 30 06:26:26.752939 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.752911 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" event={"ID":"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3","Type":"ContainerStarted","Data":"b03942f9869fbd44f59c3dd8c33400f1b85a3d30fb6d4a87bc02b9066f5bbcab"} Oct 30 06:26:26.753112 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.752960 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" event={"ID":"92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3","Type":"ContainerStarted","Data":"9c68ab09215cda5facc90318f685c1589a6f05e8ed559279bc5ca5f599087a0a"} Oct 30 06:26:26.796324 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.796262 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podStartSLOduration=1.79624765 podStartE2EDuration="1.79624765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:26.795671042 +0000 UTC m=+3346.399360299" watchObservedRunningTime="2026-10-30 06:26:26.79624765 +0000 UTC m=+3346.399936907" Oct 30 06:26:26.840906 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:26.840850 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podStartSLOduration=1.840835166 podStartE2EDuration="1.840835166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:26.840020085 +0000 UTC m=+3346.443709382" watchObservedRunningTime="2026-10-30 06:26:26.840835166 +0000 UTC m=+3346.444524423" Oct 30 06:26:27.056853 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:27.056757 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:27.058702 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:27.058672 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 06:26:27.058850 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:27.058730 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 06:26:27.098326 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:27.098296 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:27.099626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:27.099601 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 06:26:27.099747 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:27.099649 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 06:26:28.057966 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:28.057913 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 06:26:28.058672 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:28.058629 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 06:26:28.098531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:28.098491 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 06:26:28.098702 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:28.098547 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 06:26:29.056680 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:29.056638 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 06:26:29.056999 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:29.056963 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 06:26:29.098430 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:29.098404 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 06:26:29.098894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:29.098866 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 06:26:30.057983 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:30.057942 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 06:26:30.058273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:30.058244 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 06:26:30.098085 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:30.098054 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 06:26:30.098256 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:30.098108 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 06:26:30.736502 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:26:30.735792 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:26:31.059759 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:31.059655 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 06:26:31.059759 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:31.059721 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 06:26:31.097813 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:31.097784 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 06:26:31.097986 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:31.097833 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 06:26:32.058483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:32.057841 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 06:26:32.058483 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:32.057905 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 06:26:32.098436 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:32.098404 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 06:26:32.098624 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:32.098480 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 06:26:33.056583 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:33.056539 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 06:26:33.056826 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:33.056613 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 06:26:33.098528 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:33.098436 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 06:26:33.099085 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:33.099051 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 06:26:34.057081 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:34.057029 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 06:26:34.057316 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:34.057115 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 06:26:35.099698 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:35.099352 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:35.099698 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:35.099421 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:36.099661 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:36.099624 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:36.099890 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:36.099698 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:36.143484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:36.141580 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:36.143484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:36.141643 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:37.141643 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:37.141592 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:37.142330 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:37.142246 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:37.183720 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:37.183681 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:37.184036 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:37.184007 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:38.151488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:38.149586 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:38.151488 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:38.149667 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:38.186180 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:38.186133 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:38.186535 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:38.186502 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:39.201486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:39.200552 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:39.201486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:39.200619 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:39.234484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:39.233559 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:39.234484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:39.233622 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:40.243313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:40.242560 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:40.243313 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:40.242632 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:40.278633 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:40.278596 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:40.278816 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:40.278646 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:40.780592 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:40.780564 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:40.781020 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:40.780992 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:40.781663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:40.781645 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" Oct 30 06:26:41.056161 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:41.056071 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:41.056373 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:41.056351 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:26:41.057201 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:26:41.057176 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" Oct 30 06:27:00.710175 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:27:00.710084 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:27:30.710258 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:27:30.710219 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:28:00.709643 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:28:00.709601 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:28:12.718484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:28:12.717606 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-ctspw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:28:12.718484 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:28:12.717742 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-ctspw" podUID="6b24ad0f-1934-49a2-9da3-95b2c8769b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:28:30.839747 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:28:30.839702 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:29:00.710221 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:29:00.710181 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:29:30.709884 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:29:30.709838 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:29:51.494805 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.494716 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7"] Oct 30 06:29:51.498578 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.498553 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.516544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.516499 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-627222-openshift-default-dockercfg-7mm6r\"" Oct 30 06:29:51.530332 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.530302 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7"] Oct 30 06:29:51.673945 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.673896 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660855e0-7e72-4033-9780-01388d67c749-istiod-ca-cert\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.674233 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.674207 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-certs\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.675121 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.675097 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660855e0-7e72-4033-9780-01388d67c749-istio-podinfo\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.675294 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.675275 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-credential-socket\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.675486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.675446 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-istio-token\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.675722 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.675695 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zmr\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-kube-api-access-65zmr\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.675909 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.675891 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-envoy\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.676062 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.676046 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-socket\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.676245 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.676227 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-data\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.776982 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.776899 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zmr\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-kube-api-access-65zmr\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.776982 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.776946 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-envoy\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.776982 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.776978 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-socket\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777235 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777010 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-data\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777235 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777135 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660855e0-7e72-4033-9780-01388d67c749-istiod-ca-cert\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777235 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777170 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-certs\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777402 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777336 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660855e0-7e72-4033-9780-01388d67c749-istio-podinfo\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777402 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777390 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-credential-socket\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777498 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-istio-token\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777529 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-certs\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777555 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-socket\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777728 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777652 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-data\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.777841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.777817 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-credential-socket\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.778187 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.778169 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660855e0-7e72-4033-9780-01388d67c749-istiod-ca-cert\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.779547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.779522 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-envoy\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.779932 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.779907 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660855e0-7e72-4033-9780-01388d67c749-istio-podinfo\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.800260 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.800233 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-istio-token\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.800523 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.800503 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zmr\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-kube-api-access-65zmr\") pod \"e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:51.813241 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:51.813214 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:52.133242 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:52.133210 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7"] Oct 30 06:29:52.133857 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:29:52.133822 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660855e0_7e72_4033_9780_01388d67c749.slice/crio-7bce3d1eff79085dbabde9d08b26acb3ffe8ecb34a4dc26ba0d68b5f78269382 WatchSource:0}: Error finding container 7bce3d1eff79085dbabde9d08b26acb3ffe8ecb34a4dc26ba0d68b5f78269382: Status 404 returned error can't find the container with id 7bce3d1eff79085dbabde9d08b26acb3ffe8ecb34a4dc26ba0d68b5f78269382 Oct 30 06:29:52.136740 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:52.136699 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:29:52.136845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:52.136788 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:29:52.136845 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:52.136831 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:29:52.234128 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:52.234075 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" event={"ID":"660855e0-7e72-4033-9780-01388d67c749","Type":"ContainerStarted","Data":"3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470"} Oct 30 06:29:52.234315 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:52.234149 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" event={"ID":"660855e0-7e72-4033-9780-01388d67c749","Type":"ContainerStarted","Data":"7bce3d1eff79085dbabde9d08b26acb3ffe8ecb34a4dc26ba0d68b5f78269382"} Oct 30 06:29:52.279908 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:52.279835 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" podStartSLOduration=1.279820744 podStartE2EDuration="1.279820744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:52.277979567 +0000 UTC m=+3551.881668825" watchObservedRunningTime="2026-10-30 06:29:52.279820744 +0000 UTC m=+3551.883510013" Oct 30 06:29:52.813687 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:52.813656 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:53.819364 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:53.819327 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:54.247024 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:54.246984 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:29:54.248077 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:29:54.248046 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:30:00.729364 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:30:00.729324 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:30:09.248119 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:09.248039 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 06:30:09.248119 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:09.248096 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 06:30:09.331451 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:09.331370 5072 generic.go:356] "Generic (PLEG): container finished" podID="660855e0-7e72-4033-9780-01388d67c749" containerID="3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470" exitCode=0 Oct 30 06:30:09.331451 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:09.331428 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" event={"ID":"660855e0-7e72-4033-9780-01388d67c749","Type":"ContainerDied","Data":"3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470"} Oct 30 06:30:09.331857 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:09.331841 5072 scope.go:122] "RemoveContainer" containerID="3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470" Oct 30 06:30:09.333116 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:09.333087 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:30:09.333210 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:09.333157 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:30:09.333210 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:09.333190 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:30:10.341249 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:10.341212 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" event={"ID":"660855e0-7e72-4033-9780-01388d67c749","Type":"ContainerStarted","Data":"3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef"} Oct 30 06:30:10.822575 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:10.822542 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:30:10.823022 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:10.823000 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 06:30:10.823199 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:10.823174 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 06:30:11.828485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:11.821938 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 06:30:11.828485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:11.821998 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 06:30:12.821485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:12.814837 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 06:30:12.821485 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:12.814898 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 06:30:13.820549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:13.818933 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 06:30:13.820549 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:13.819004 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 06:30:14.813911 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:14.813873 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 06:30:14.814229 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:14.814197 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 06:30:16.027533 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:16.027501 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:30:16.028769 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:16.028318 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:30:16.029516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:16.029495 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:30:30.712071 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:30:30.712024 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:30:48.663304 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:48.663271 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:30:48.666810 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:30:48.666789 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:31:00.710316 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:31:00.710278 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:31:06.847539 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:06.847496 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7"] Oct 30 06:31:06.848266 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:06.847780 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" containerID="cri-o://3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef" gracePeriod=30 Oct 30 06:31:12.107666 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.107642 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:31:12.230921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.230836 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-data\") pod \"660855e0-7e72-4033-9780-01388d67c749\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " Oct 30 06:31:12.230921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.230902 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-credential-socket\") pod \"660855e0-7e72-4033-9780-01388d67c749\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " Oct 30 06:31:12.231156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.230935 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-kube-api-access-65zmr\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-kube-api-access-65zmr\") pod \"660855e0-7e72-4033-9780-01388d67c749\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " Oct 30 06:31:12.231156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.230973 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-istio-token\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-istio-token\") pod \"660855e0-7e72-4033-9780-01388d67c749\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " Oct 30 06:31:12.231156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231013 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660855e0-7e72-4033-9780-01388d67c749-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660855e0-7e72-4033-9780-01388d67c749-istiod-ca-cert\") pod \"660855e0-7e72-4033-9780-01388d67c749\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " Oct 30 06:31:12.231156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231119 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/660855e0-7e72-4033-9780-01388d67c749-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660855e0-7e72-4033-9780-01388d67c749-istio-podinfo\") pod \"660855e0-7e72-4033-9780-01388d67c749\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " Oct 30 06:31:12.231156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231131 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-data" pod "660855e0-7e72-4033-9780-01388d67c749" (UID: "660855e0-7e72-4033-9780-01388d67c749"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:12.231156 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231146 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-socket\") pod \"660855e0-7e72-4033-9780-01388d67c749\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " Oct 30 06:31:12.231524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231186 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-certs\") pod \"660855e0-7e72-4033-9780-01388d67c749\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " Oct 30 06:31:12.231524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231188 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-credential-socket" pod "660855e0-7e72-4033-9780-01388d67c749" (UID: "660855e0-7e72-4033-9780-01388d67c749"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:12.231524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231212 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-envoy\") pod \"660855e0-7e72-4033-9780-01388d67c749\" (UID: \"660855e0-7e72-4033-9780-01388d67c749\") " Oct 30 06:31:12.231524 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231430 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-socket" pod "660855e0-7e72-4033-9780-01388d67c749" (UID: "660855e0-7e72-4033-9780-01388d67c749"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:12.231727 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231531 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660855e0-7e72-4033-9780-01388d67c749-istiod-ca-cert" pod "660855e0-7e72-4033-9780-01388d67c749" (UID: "660855e0-7e72-4033-9780-01388d67c749"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:12.231727 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231597 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-certs" pod "660855e0-7e72-4033-9780-01388d67c749" (UID: "660855e0-7e72-4033-9780-01388d67c749"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:12.231799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231725 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.231799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231746 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.231799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231760 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.231799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231771 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.231799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.231783 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660855e0-7e72-4033-9780-01388d67c749-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.233616 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.233592 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-envoy" pod "660855e0-7e72-4033-9780-01388d67c749" (UID: "660855e0-7e72-4033-9780-01388d67c749"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:12.233733 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.233618 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-istio-token" pod "660855e0-7e72-4033-9780-01388d67c749" (UID: "660855e0-7e72-4033-9780-01388d67c749"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:12.233868 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.233846 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-kube-api-access-65zmr" pod "660855e0-7e72-4033-9780-01388d67c749" (UID: "660855e0-7e72-4033-9780-01388d67c749"). InnerVolumeSpecName "kube-api-access-65zmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:12.233868 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.233850 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/660855e0-7e72-4033-9780-01388d67c749-istio-podinfo" pod "660855e0-7e72-4033-9780-01388d67c749" (UID: "660855e0-7e72-4033-9780-01388d67c749"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:31:12.333067 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.333023 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.333067 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.333064 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660855e0-7e72-4033-9780-01388d67c749-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.333067 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.333073 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660855e0-7e72-4033-9780-01388d67c749-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.333067 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.333082 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65zmr\" (UniqueName: \"kubernetes.io/projected/660855e0-7e72-4033-9780-01388d67c749-kube-api-access-65zmr\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.684245 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.684213 5072 generic.go:356] "Generic (PLEG): container finished" podID="660855e0-7e72-4033-9780-01388d67c749" containerID="3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef" exitCode=0 Oct 30 06:31:12.684427 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.684274 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" Oct 30 06:31:12.684427 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.684301 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" event={"ID":"660855e0-7e72-4033-9780-01388d67c749","Type":"ContainerDied","Data":"3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef"} Oct 30 06:31:12.684427 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.684378 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7" event={"ID":"660855e0-7e72-4033-9780-01388d67c749","Type":"ContainerDied","Data":"7bce3d1eff79085dbabde9d08b26acb3ffe8ecb34a4dc26ba0d68b5f78269382"} Oct 30 06:31:12.684427 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.684401 5072 scope.go:122] "RemoveContainer" containerID="3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef" Oct 30 06:31:12.694295 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.694274 5072 scope.go:122] "RemoveContainer" containerID="3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470" Oct 30 06:31:12.703855 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.703836 5072 scope.go:122] "RemoveContainer" containerID="3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef" Oct 30 06:31:12.704149 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:31:12.704130 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef\": container with ID starting with 3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef not found: ID does not exist" containerID="3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef" Oct 30 06:31:12.704239 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.704172 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef"} err="failed to get container status \"3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef\": rpc error: code = NotFound desc = could not find container \"3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef\": container with ID starting with 3bc88d82b1fac44c8029d2d91f5ae5dffab5cac8400bf0383e92fd0bca9e8bef not found: ID does not exist" Oct 30 06:31:12.704239 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.704207 5072 scope.go:122] "RemoveContainer" containerID="3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470" Oct 30 06:31:12.704527 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:31:12.704500 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470\": container with ID starting with 3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470 not found: ID does not exist" containerID="3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470" Oct 30 06:31:12.704640 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.704531 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470"} err="failed to get container status \"3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470\": rpc error: code = NotFound desc = could not find container \"3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470\": container with ID starting with 3944ca5f4557d3cb613ef8a40effc75644140956d0ea52399d71ba613028e470 not found: ID does not exist" Oct 30 06:31:12.748794 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.748764 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7"] Oct 30 06:31:12.761037 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:12.761008 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-627222-openshift-default-77d7c5b59cntzw7"] Oct 30 06:31:14.517607 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:14.517576 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660855e0-7e72-4033-9780-01388d67c749" path="/var/lib/kubelet/pods/660855e0-7e72-4033-9780-01388d67c749/volumes" Oct 30 06:31:30.708832 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:31:30.708794 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:31:32.322116 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.322074 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv"] Oct 30 06:31:32.322597 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.322266 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" podUID="337e2886-d4e4-4944-abcf-dcea923a1e7b" containerName="manager" containerID="cri-o://b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55" gracePeriod=10 Oct 30 06:31:32.573546 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.573488 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 06:31:32.706708 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.706676 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337e2886-d4e4-4944-abcf-dcea923a1e7b-kube-api-access-87mvw\" (UniqueName: \"kubernetes.io/projected/337e2886-d4e4-4944-abcf-dcea923a1e7b-kube-api-access-87mvw\") pod \"337e2886-d4e4-4944-abcf-dcea923a1e7b\" (UID: \"337e2886-d4e4-4944-abcf-dcea923a1e7b\") " Oct 30 06:31:32.706905 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.706748 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337e2886-d4e4-4944-abcf-dcea923a1e7b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/337e2886-d4e4-4944-abcf-dcea923a1e7b-extensions-socket-volume\") pod \"337e2886-d4e4-4944-abcf-dcea923a1e7b\" (UID: \"337e2886-d4e4-4944-abcf-dcea923a1e7b\") " Oct 30 06:31:32.707168 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.707140 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337e2886-d4e4-4944-abcf-dcea923a1e7b-extensions-socket-volume" pod "337e2886-d4e4-4944-abcf-dcea923a1e7b" (UID: "337e2886-d4e4-4944-abcf-dcea923a1e7b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:32.709047 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.709025 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337e2886-d4e4-4944-abcf-dcea923a1e7b-kube-api-access-87mvw" pod "337e2886-d4e4-4944-abcf-dcea923a1e7b" (UID: "337e2886-d4e4-4944-abcf-dcea923a1e7b"). InnerVolumeSpecName "kube-api-access-87mvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:32.808626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.808593 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87mvw\" (UniqueName: \"kubernetes.io/projected/337e2886-d4e4-4944-abcf-dcea923a1e7b-kube-api-access-87mvw\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:32.808626 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.808622 5072 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/337e2886-d4e4-4944-abcf-dcea923a1e7b-extensions-socket-volume\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:31:32.841419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.841334 5072 generic.go:356] "Generic (PLEG): container finished" podID="337e2886-d4e4-4944-abcf-dcea923a1e7b" containerID="b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55" exitCode=0 Oct 30 06:31:32.841419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.841376 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" event={"ID":"337e2886-d4e4-4944-abcf-dcea923a1e7b","Type":"ContainerDied","Data":"b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55"} Oct 30 06:31:32.841419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.841383 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" Oct 30 06:31:32.841724 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.841430 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv" event={"ID":"337e2886-d4e4-4944-abcf-dcea923a1e7b","Type":"ContainerDied","Data":"4d223c9b1d30ef70f3b21063c80c7914cd8cb0bc3e11f7dcfd826bc192d9ce6b"} Oct 30 06:31:32.841724 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.841491 5072 scope.go:122] "RemoveContainer" containerID="b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55" Oct 30 06:31:32.852053 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.852034 5072 scope.go:122] "RemoveContainer" containerID="b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55" Oct 30 06:31:32.852321 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:31:32.852305 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55\": container with ID starting with b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55 not found: ID does not exist" containerID="b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55" Oct 30 06:31:32.852403 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.852328 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55"} err="failed to get container status \"b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55\": rpc error: code = NotFound desc = could not find container \"b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55\": container with ID starting with b00e2f499d6b521b89ed58448575caec76e06d62c3c243c4d1c115d044f05b55 not found: ID does not exist" Oct 30 06:31:32.893633 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.893598 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv"] Oct 30 06:31:32.905875 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:32.905846 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4c4mv"] Oct 30 06:31:34.516858 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:31:34.516826 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337e2886-d4e4-4944-abcf-dcea923a1e7b" path="/var/lib/kubelet/pods/337e2886-d4e4-4944-abcf-dcea923a1e7b/volumes" Oct 30 06:32:00.709171 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:32:00.709132 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:32:30.709730 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:32:30.709691 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:32:35.464938 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.464887 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk"] Oct 30 06:32:35.465354 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.465341 5072 cpu_manager.go:414] "containerMap: removing container" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" Oct 30 06:32:35.465399 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.465355 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" Oct 30 06:32:35.465399 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.465375 5072 cpu_manager.go:414] "containerMap: removing container" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" Oct 30 06:32:35.465399 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.465380 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" Oct 30 06:32:35.465399 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.465399 5072 cpu_manager.go:414] "containerMap: removing container" podUID="337e2886-d4e4-4944-abcf-dcea923a1e7b" containerName="manager" Oct 30 06:32:35.465563 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.465404 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337e2886-d4e4-4944-abcf-dcea923a1e7b" containerName="manager" Oct 30 06:32:35.465563 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.465506 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" Oct 30 06:32:35.465563 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.465521 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="660855e0-7e72-4033-9780-01388d67c749" containerName="istio-proxy" Oct 30 06:32:35.465563 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.465536 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="337e2886-d4e4-4944-abcf-dcea923a1e7b" containerName="manager" Oct 30 06:32:35.468774 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.468748 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:35.485715 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.485688 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk"] Oct 30 06:32:35.495274 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.495253 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2lbtb\"" Oct 30 06:32:35.552034 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.552000 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksgf\" (UniqueName: \"kubernetes.io/projected/26f8e1e4-b502-4faf-bbdc-3e0774e9021b-kube-api-access-mksgf\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rwfsk\" (UID: \"26f8e1e4-b502-4faf-bbdc-3e0774e9021b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:35.552209 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.552037 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26f8e1e4-b502-4faf-bbdc-3e0774e9021b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rwfsk\" (UID: \"26f8e1e4-b502-4faf-bbdc-3e0774e9021b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:35.653136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.653101 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksgf\" (UniqueName: \"kubernetes.io/projected/26f8e1e4-b502-4faf-bbdc-3e0774e9021b-kube-api-access-mksgf\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rwfsk\" (UID: \"26f8e1e4-b502-4faf-bbdc-3e0774e9021b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:35.653136 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.653135 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26f8e1e4-b502-4faf-bbdc-3e0774e9021b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rwfsk\" (UID: \"26f8e1e4-b502-4faf-bbdc-3e0774e9021b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:35.653564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.653548 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26f8e1e4-b502-4faf-bbdc-3e0774e9021b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rwfsk\" (UID: \"26f8e1e4-b502-4faf-bbdc-3e0774e9021b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:35.676563 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.676537 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksgf\" (UniqueName: \"kubernetes.io/projected/26f8e1e4-b502-4faf-bbdc-3e0774e9021b-kube-api-access-mksgf\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rwfsk\" (UID: \"26f8e1e4-b502-4faf-bbdc-3e0774e9021b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:35.782265 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:35.782188 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:36.109074 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:36.109045 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk"] Oct 30 06:32:36.110138 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:32:36.110112 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f8e1e4_b502_4faf_bbdc_3e0774e9021b.slice/crio-d5bbb373c659f3e7f179b3585f9a619ea92889abe1d141b767c49ca8daca0e64 WatchSource:0}: Error finding container d5bbb373c659f3e7f179b3585f9a619ea92889abe1d141b767c49ca8daca0e64: Status 404 returned error can't find the container with id d5bbb373c659f3e7f179b3585f9a619ea92889abe1d141b767c49ca8daca0e64 Oct 30 06:32:36.166704 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:36.166669 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" event={"ID":"26f8e1e4-b502-4faf-bbdc-3e0774e9021b","Type":"ContainerStarted","Data":"d5bbb373c659f3e7f179b3585f9a619ea92889abe1d141b767c49ca8daca0e64"} Oct 30 06:32:37.173218 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:37.173177 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" event={"ID":"26f8e1e4-b502-4faf-bbdc-3e0774e9021b","Type":"ContainerStarted","Data":"de42b53350ddb1fe4332a0f3b4d48fa2b416d8e996eabb569cd72e8606ff3055"} Oct 30 06:32:37.173715 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:37.173375 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:37.246400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:37.246342 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" podStartSLOduration=2.246330173 podStartE2EDuration="2.246330173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:37.233039173 +0000 UTC m=+3716.836728429" watchObservedRunningTime="2026-10-30 06:32:37.246330173 +0000 UTC m=+3716.850019429" Oct 30 06:32:48.180951 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:48.180919 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rwfsk" Oct 30 06:32:56.837672 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:56.837549 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-7lvmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:32:56.838326 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:56.838272 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-7lvmn" podUID="9862ec03-2fe1-42dc-8f6b-305348cc8bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:32:57.116482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:57.114575 5072 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-q5bls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:32:57.116482 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:32:57.114672 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-q5bls" podUID="92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:33:00.709037 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:33:00.708996 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:33:27.473517 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.473478 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f"] Oct 30 06:33:27.477957 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.477930 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.492897 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.492869 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-be81b5-openshift-default-dockercfg-gwlzh\"" Oct 30 06:33:27.507516 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.507485 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f"] Oct 30 06:33:27.666916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.659020 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-data\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.666916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.659130 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-token\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.666916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.659167 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxqj\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-kube-api-access-lcxqj\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.666916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.659220 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-envoy\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.666916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.659248 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-certs\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.666916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.659304 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-socket\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.666916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.659343 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-podinfo\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.666916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.659424 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-credential-socket\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.666916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.659484 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istiod-ca-cert\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.760921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.760839 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-socket\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.760921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.760884 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-podinfo\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.760921 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.760923 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-credential-socket\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.760951 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istiod-ca-cert\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.760981 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-data\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.761055 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-token\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.761094 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxqj\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-kube-api-access-lcxqj\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.761135 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-envoy\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.761165 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-certs\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.761357 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-credential-socket\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.761401 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-socket\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.761438 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-data\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761602 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.761574 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-certs\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.761901 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.761884 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istiod-ca-cert\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.763703 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.763679 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-envoy\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.763797 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.763713 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-podinfo\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.784590 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.784565 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-token\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.784737 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.784719 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxqj\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-kube-api-access-lcxqj\") pod \"e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:27.796541 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:27.796521 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:28.116550 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:28.116519 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f"] Oct 30 06:33:28.117593 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:33:28.117566 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4bd4e3_19d2_4626_853d_f7c41c44eb48.slice/crio-9e9fc68f0132e80e2b1aa5d1c38a219c92c82a4b36380ace6f2f138b1f786f1c WatchSource:0}: Error finding container 9e9fc68f0132e80e2b1aa5d1c38a219c92c82a4b36380ace6f2f138b1f786f1c: Status 404 returned error can't find the container with id 9e9fc68f0132e80e2b1aa5d1c38a219c92c82a4b36380ace6f2f138b1f786f1c Oct 30 06:33:28.119905 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:28.119867 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:33:28.120003 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:28.119941 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:33:28.120003 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:28.119973 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:33:28.507229 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:28.507136 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" event={"ID":"3b4bd4e3-19d2-4626-853d-f7c41c44eb48","Type":"ContainerStarted","Data":"d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a"} Oct 30 06:33:28.507663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:28.507236 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" event={"ID":"3b4bd4e3-19d2-4626-853d-f7c41c44eb48","Type":"ContainerStarted","Data":"9e9fc68f0132e80e2b1aa5d1c38a219c92c82a4b36380ace6f2f138b1f786f1c"} Oct 30 06:33:28.564234 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:28.564183 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" podStartSLOduration=1.5641711090000001 podStartE2EDuration="1.564171109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:28.563190024 +0000 UTC m=+3768.166879280" watchObservedRunningTime="2026-10-30 06:33:28.564171109 +0000 UTC m=+3768.167860367" Oct 30 06:33:28.797106 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:28.797021 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:29.802772 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:29.802742 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:30.518246 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:30.518216 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:30.518425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:30.518300 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:30.718874 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:33:30.718837 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:33:46.369762 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:46.369723 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": read tcp 10.132.0.2:60698->10.132.0.223:15021: read: connection reset by peer" start-of-body= Oct 30 06:33:46.370224 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:46.369800 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": read tcp 10.132.0.2:60698->10.132.0.223:15021: read: connection reset by peer" Oct 30 06:33:46.610447 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:46.610414 5072 generic.go:356] "Generic (PLEG): container finished" podID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerID="d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a" exitCode=0 Oct 30 06:33:46.610628 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:46.610494 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" event={"ID":"3b4bd4e3-19d2-4626-853d-f7c41c44eb48","Type":"ContainerDied","Data":"d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a"} Oct 30 06:33:46.610901 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:46.610888 5072 scope.go:122] "RemoveContainer" containerID="d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a" Oct 30 06:33:46.612152 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:46.612122 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:33:46.612238 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:46.612220 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:33:46.612292 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:46.612276 5072 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 06:33:47.639400 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:47.639367 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" event={"ID":"3b4bd4e3-19d2-4626-853d-f7c41c44eb48","Type":"ContainerStarted","Data":"656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143"} Oct 30 06:33:47.796881 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:47.796844 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:47.797218 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:47.797195 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 06:33:47.797267 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:47.797249 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 06:33:48.797552 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:48.797518 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 06:33:48.797962 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:48.797569 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 06:33:49.797450 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:49.797412 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 06:33:49.797695 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:49.797475 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 06:33:50.797941 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:50.797904 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 06:33:50.798370 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:50.797975 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 06:33:51.796977 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:51.796941 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 06:33:51.797168 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:51.797006 5072 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 06:33:53.246321 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:53.246287 5072 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:53.246873 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:53.246713 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:33:53.248041 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:33:53.248013 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:34:00.709977 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:34:00.709939 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:34:30.710646 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:34:30.710607 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:34:36.976894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:36.976852 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f"] Oct 30 06:34:36.977595 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:36.977117 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" containerID="cri-o://656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143" gracePeriod=30 Oct 30 06:34:38.248956 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:38.248922 5072 patch_prober.go:29] interesting pod/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:34:38.249346 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:38.249000 5072 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:34:42.246935 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.246865 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:34:42.361564 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361524 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-envoy\") pod \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " Oct 30 06:34:42.361772 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361597 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-kube-api-access-lcxqj\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-kube-api-access-lcxqj\") pod \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " Oct 30 06:34:42.361772 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361633 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-socket\") pod \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " Oct 30 06:34:42.361772 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361680 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-podinfo\") pod \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " Oct 30 06:34:42.361772 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361752 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-token\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-token\") pod \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " Oct 30 06:34:42.362014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361791 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-credential-socket\") pod \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " Oct 30 06:34:42.362014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361869 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istiod-ca-cert\") pod \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " Oct 30 06:34:42.362014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361906 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-data\") pod \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " Oct 30 06:34:42.362014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361950 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-certs\") pod \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\" (UID: \"3b4bd4e3-19d2-4626-853d-f7c41c44eb48\") " Oct 30 06:34:42.362014 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.361945 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-socket" pod "3b4bd4e3-19d2-4626-853d-f7c41c44eb48" (UID: "3b4bd4e3-19d2-4626-853d-f7c41c44eb48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:34:42.362271 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.362050 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-credential-socket" pod "3b4bd4e3-19d2-4626-853d-f7c41c44eb48" (UID: "3b4bd4e3-19d2-4626-853d-f7c41c44eb48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:34:42.362271 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.362184 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-data" pod "3b4bd4e3-19d2-4626-853d-f7c41c44eb48" (UID: "3b4bd4e3-19d2-4626-853d-f7c41c44eb48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:34:42.362371 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.362268 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istiod-ca-cert" pod "3b4bd4e3-19d2-4626-853d-f7c41c44eb48" (UID: "3b4bd4e3-19d2-4626-853d-f7c41c44eb48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:34:42.362416 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.362361 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-certs" pod "3b4bd4e3-19d2-4626-853d-f7c41c44eb48" (UID: "3b4bd4e3-19d2-4626-853d-f7c41c44eb48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:34:42.362489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.362423 5072 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:34:42.362489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.362438 5072 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-credential-socket\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:34:42.362489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.362448 5072 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istiod-ca-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:34:42.362489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.362482 5072 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-data\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:34:42.364302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.364275 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-envoy" pod "3b4bd4e3-19d2-4626-853d-f7c41c44eb48" (UID: "3b4bd4e3-19d2-4626-853d-f7c41c44eb48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:34:42.364302 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.364291 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-token" pod "3b4bd4e3-19d2-4626-853d-f7c41c44eb48" (UID: "3b4bd4e3-19d2-4626-853d-f7c41c44eb48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:34:42.364622 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.364598 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-podinfo" pod "3b4bd4e3-19d2-4626-853d-f7c41c44eb48" (UID: "3b4bd4e3-19d2-4626-853d-f7c41c44eb48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:34:42.364723 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.364685 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-kube-api-access-lcxqj" pod "3b4bd4e3-19d2-4626-853d-f7c41c44eb48" (UID: "3b4bd4e3-19d2-4626-853d-f7c41c44eb48"). InnerVolumeSpecName "kube-api-access-lcxqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:34:42.464076 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.464040 5072 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-workload-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:34:42.464076 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.464072 5072 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-envoy\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:34:42.464076 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.464081 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcxqj\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-kube-api-access-lcxqj\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:34:42.464357 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.464091 5072 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-podinfo\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:34:42.464357 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.464100 5072 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4bd4e3-19d2-4626-853d-f7c41c44eb48-istio-token\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:34:42.935296 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.935259 5072 generic.go:356] "Generic (PLEG): container finished" podID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerID="656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143" exitCode=0 Oct 30 06:34:42.935560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.935313 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" Oct 30 06:34:42.935560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.935332 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" event={"ID":"3b4bd4e3-19d2-4626-853d-f7c41c44eb48","Type":"ContainerDied","Data":"656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143"} Oct 30 06:34:42.935560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.935385 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f" event={"ID":"3b4bd4e3-19d2-4626-853d-f7c41c44eb48","Type":"ContainerDied","Data":"9e9fc68f0132e80e2b1aa5d1c38a219c92c82a4b36380ace6f2f138b1f786f1c"} Oct 30 06:34:42.935560 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.935414 5072 scope.go:122] "RemoveContainer" containerID="656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143" Oct 30 06:34:42.950547 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.950527 5072 scope.go:122] "RemoveContainer" containerID="d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a" Oct 30 06:34:42.959562 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.959540 5072 scope.go:122] "RemoveContainer" containerID="656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143" Oct 30 06:34:42.959905 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:34:42.959888 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143\": container with ID starting with 656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143 not found: ID does not exist" containerID="656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143" Oct 30 06:34:42.959990 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.959918 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143"} err="failed to get container status \"656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143\": rpc error: code = NotFound desc = could not find container \"656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143\": container with ID starting with 656eb03df62f328962f15e43a4731d4881b5e5c0a7e9933d52cedaac9d187143 not found: ID does not exist" Oct 30 06:34:42.959990 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.959946 5072 scope.go:122] "RemoveContainer" containerID="d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a" Oct 30 06:34:42.960227 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:34:42.960210 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a\": container with ID starting with d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a not found: ID does not exist" containerID="d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a" Oct 30 06:34:42.960285 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:42.960237 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a"} err="failed to get container status \"d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a\": rpc error: code = NotFound desc = could not find container \"d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a\": container with ID starting with d7cb27db7571bc94fdebe33b8968964d0b17a2f86cfc81b4e239271151cc2d2a not found: ID does not exist" Oct 30 06:34:43.000663 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:43.000634 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f"] Oct 30 06:34:43.012833 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:43.012804 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-be81b5-openshift-default-5bb5c89669vv76f"] Oct 30 06:34:44.517923 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:34:44.517891 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" path="/var/lib/kubelet/pods/3b4bd4e3-19d2-4626-853d-f7c41c44eb48/volumes" Oct 30 06:35:00.717056 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:35:00.716960 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:35:30.730171 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:35:30.730130 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:35:48.703801 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:35:48.703768 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:35:48.704682 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:35:48.704656 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:36:00.709412 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:36:00.709332 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:36:10.544576 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.544536 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-qhq6q"] Oct 30 06:36:10.545350 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.545325 5072 cpu_manager.go:414] "containerMap: removing container" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" Oct 30 06:36:10.545402 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.545356 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" Oct 30 06:36:10.545614 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.545594 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" Oct 30 06:36:10.549704 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.549680 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.569100 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.568836 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-qhq6q"] Oct 30 06:36:10.743012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.742975 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc852086-cc8f-497c-aa74-962466cdd08e-maas-api-tls\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.743185 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.743021 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8km7\" (UniqueName: \"kubernetes.io/projected/cc852086-cc8f-497c-aa74-962466cdd08e-kube-api-access-x8km7\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.743185 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.743051 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc852086-cc8f-497c-aa74-962466cdd08e-metrics-certs\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.743285 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.743258 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc852086-cc8f-497c-aa74-962466cdd08e-openshift-service-ca\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.844251 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.844138 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc852086-cc8f-497c-aa74-962466cdd08e-metrics-certs\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.844429 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.844252 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc852086-cc8f-497c-aa74-962466cdd08e-openshift-service-ca\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.844429 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.844340 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc852086-cc8f-497c-aa74-962466cdd08e-maas-api-tls\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.844429 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.844379 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8km7\" (UniqueName: \"kubernetes.io/projected/cc852086-cc8f-497c-aa74-962466cdd08e-kube-api-access-x8km7\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.845258 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.845228 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc852086-cc8f-497c-aa74-962466cdd08e-openshift-service-ca\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.847037 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.847009 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc852086-cc8f-497c-aa74-962466cdd08e-metrics-certs\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.847144 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.847096 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc852086-cc8f-497c-aa74-962466cdd08e-maas-api-tls\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:10.867489 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:10.867438 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8km7\" (UniqueName: \"kubernetes.io/projected/cc852086-cc8f-497c-aa74-962466cdd08e-kube-api-access-x8km7\") pod \"maas-api-595cc648c-qhq6q\" (UID: \"cc852086-cc8f-497c-aa74-962466cdd08e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:11.164401 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:11.164364 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:11.464134 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:11.464110 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-qhq6q"] Oct 30 06:36:11.465060 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:36:11.465030 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc852086_cc8f_497c_aa74_962466cdd08e.slice/crio-635aa9c7e6c0ccd10ad7a6ee326e4d8082dc5768f77335f78e782baa34008b20 WatchSource:0}: Error finding container 635aa9c7e6c0ccd10ad7a6ee326e4d8082dc5768f77335f78e782baa34008b20: Status 404 returned error can't find the container with id 635aa9c7e6c0ccd10ad7a6ee326e4d8082dc5768f77335f78e782baa34008b20 Oct 30 06:36:11.466487 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:11.466441 5072 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:36:11.662919 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:11.662875 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" event={"ID":"cc852086-cc8f-497c-aa74-962466cdd08e","Type":"ContainerStarted","Data":"635aa9c7e6c0ccd10ad7a6ee326e4d8082dc5768f77335f78e782baa34008b20"} Oct 30 06:36:12.670605 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:12.670563 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" event={"ID":"cc852086-cc8f-497c-aa74-962466cdd08e","Type":"ContainerStarted","Data":"b647e23ee509604b9a9763220f875dea16f7e7ca303795c80c06e33336cee257"} Oct 30 06:36:12.671041 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:12.670637 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:12.714151 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:12.714098 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" podStartSLOduration=2.409300309 podStartE2EDuration="2.714084382s" totalImagesPullingTime="304.784073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:12.712358339 +0000 UTC m=+3932.316047595" watchObservedRunningTime="2026-10-30 06:36:12.714084382 +0000 UTC m=+3932.317773639" Oct 30 06:36:18.682587 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:18.682555 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-qhq6q" Oct 30 06:36:18.758169 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:18.758128 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-tttnt"] Oct 30 06:36:18.758435 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:18.758407 5072 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" podUID="0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" containerName="maas-api" containerID="cri-o://1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75" gracePeriod=30 Oct 30 06:36:20.139173 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.139136 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 06:36:20.253022 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.252981 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-openshift-service-ca\") pod \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " Oct 30 06:36:20.253022 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.253024 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-maas-api-tls\") pod \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " Oct 30 06:36:20.253321 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.253062 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-metrics-certs\") pod \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " Oct 30 06:36:20.253321 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.253151 5072 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-kube-api-access-55fbc\" (UniqueName: \"kubernetes.io/projected/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-kube-api-access-55fbc\") pod \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\" (UID: \"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150\") " Oct 30 06:36:20.253496 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.253425 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-openshift-service-ca" pod "0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" (UID: "0a7765d7-7dd3-43b1-8f9c-4a7d293cc150"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:20.253577 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.253566 5072 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-openshift-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:36:20.256315 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.256288 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-maas-api-tls" pod "0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" (UID: "0a7765d7-7dd3-43b1-8f9c-4a7d293cc150"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:36:20.256474 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.256323 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-kube-api-access-55fbc" pod "0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" (UID: "0a7765d7-7dd3-43b1-8f9c-4a7d293cc150"). InnerVolumeSpecName "kube-api-access-55fbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:20.257031 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.257002 5072 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-metrics-certs" pod "0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" (UID: "0a7765d7-7dd3-43b1-8f9c-4a7d293cc150"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:36:20.354668 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.354625 5072 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-maas-api-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:36:20.354668 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.354665 5072 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-metrics-certs\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:36:20.354958 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.354682 5072 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55fbc\" (UniqueName: \"kubernetes.io/projected/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150-kube-api-access-55fbc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\"" Oct 30 06:36:20.772082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.753107 5072 generic.go:356] "Generic (PLEG): container finished" podID="0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" containerID="1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75" exitCode=0 Oct 30 06:36:20.772082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.753227 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" event={"ID":"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150","Type":"ContainerDied","Data":"1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75"} Oct 30 06:36:20.772082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.753279 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" event={"ID":"0a7765d7-7dd3-43b1-8f9c-4a7d293cc150","Type":"ContainerDied","Data":"6df1c2aa9e214817c278f9278bfcfd36abdfc7f9dcea0e42e00d33ccc118d8c8"} Oct 30 06:36:20.772082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.753309 5072 scope.go:122] "RemoveContainer" containerID="1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75" Oct 30 06:36:20.772082 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.753494 5072 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-tttnt" Oct 30 06:36:20.880392 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.880287 5072 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-tttnt"] Oct 30 06:36:20.880392 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.880354 5072 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-tttnt"] Oct 30 06:36:20.882651 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.882540 5072 scope.go:122] "RemoveContainer" containerID="1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75" Oct 30 06:36:20.886659 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:36:20.886578 5072 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75\": container with ID starting with 1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75 not found: ID does not exist" containerID="1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75" Oct 30 06:36:20.886659 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:20.886621 5072 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75"} err="failed to get container status \"1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75\": rpc error: code = NotFound desc = could not find container \"1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75\": container with ID starting with 1957e135132b574801e1e6cae81fa273a271ac1b458a4f81022940db81f0cb75 not found: ID does not exist" Oct 30 06:36:22.521368 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:36:22.521328 5072 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" path="/var/lib/kubelet/pods/0a7765d7-7dd3-43b1-8f9c-4a7d293cc150/volumes" Oct 30 06:36:30.726647 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:36:30.726601 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:37:00.709417 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:37:00.709380 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:37:20.550369 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:20.550293 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-5rv7s_8fa1464f-f5fc-4013-b65e-9ad8ebd4188d/manager/0.log" Oct 30 06:37:21.069799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:21.069768 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b97b5857-55xsq_de98d103-370c-4da2-8244-500a482f16a1/manager/0.log" Oct 30 06:37:22.452677 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:22.452646 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-whrp8_ed7b64b6-6071-4376-aaf5-38a2baa4c70f/copy-manifests/0.log" Oct 30 06:37:22.495916 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:22.495885 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-whrp8_ed7b64b6-6071-4376-aaf5-38a2baa4c70f/manager/0.log" Oct 30 06:37:22.741789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:22.741709 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-b6gpt_f59ca931-b8ac-4755-b89e-d06668fc00e2/proxy/0.log" Oct 30 06:37:23.506782 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:23.506745 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-qhq6q_cc852086-cc8f-497c-aa74-962466cdd08e/maas-api/0.log" Oct 30 06:37:27.487815 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:27.487785 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-fjf2l_f1802508-d7a4-48d0-b5dc-89a480de16df/manager/0.log" Oct 30 06:37:27.906020 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:27.905993 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-rwfsk_26f8e1e4-b502-4faf-bbdc-3e0774e9021b/manager/0.log" Oct 30 06:37:28.390021 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:28.389986 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-qh68j_3d979a36-0e1f-45eb-8fef-34fdecd4ba8d/manager/0.log" Oct 30 06:37:29.381034 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:29.381000 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-djzrr_91613cf9-4b13-4b1a-ac74-90ec57a6eab5/discovery/0.log" Oct 30 06:37:29.778924 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:29.778848 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788bd9946-g8t5n_14e11f60-a1b9-4a63-bb32-60f3388808f8/kube-auth-proxy/0.log" Oct 30 06:37:30.385894 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:30.385864 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-7lvmn_9862ec03-2fe1-42dc-8f6b-305348cc8bae/istio-proxy/0.log" Oct 30 06:37:30.709193 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:37:30.709113 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:37:30.799025 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:30.798994 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-ctspw_6b24ad0f-1934-49a2-9da3-95b2c8769b2f/istio-proxy/0.log" Oct 30 06:37:31.006517 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:31.006416 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-plrsv_a188fbf5-5b98-49b8-a241-11e0e437ee1a/istio-proxy/0.log" Oct 30 06:37:31.189896 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:31.189870 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-q5bls_92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3/istio-proxy/0.log" Oct 30 06:37:31.958445 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:31.958412 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xphj8_94eb79a6-731e-4990-abfc-3c9823d595fe/payload-pre-processing/0.log" Oct 30 06:37:32.338842 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:32.338750 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54dbcc76d9-x7c5h_6381b6d8-a45f-43d3-955c-504694db87da/router/0.log" Oct 30 06:37:33.121133 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:33.121104 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s_3eb1ca35-0683-4a76-ac18-5fd88ae57463/main/0.log" Oct 30 06:37:33.320310 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:33.320280 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-jvmq9_df4f6519-1409-4d13-b154-a1313e04c7fd/main/0.log" Oct 30 06:37:33.516273 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:33.516194 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-b7r6n_36b1cfca-d939-4a07-92cc-b5bc2520337b/main/0.log" Oct 30 06:37:33.909401 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:33.909369 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc_3ceabc67-51ef-4f1c-86dd-88fb31892d7f/main/0.log" Oct 30 06:37:44.588232 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:44.588201 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-5rv7s_8fa1464f-f5fc-4013-b65e-9ad8ebd4188d/manager/0.log" Oct 30 06:37:45.044531 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:45.044494 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b97b5857-55xsq_de98d103-370c-4da2-8244-500a482f16a1/manager/0.log" Oct 30 06:37:46.363643 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:46.363610 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-whrp8_ed7b64b6-6071-4376-aaf5-38a2baa4c70f/manager/0.log" Oct 30 06:37:46.428327 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:46.428300 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-whrp8_ed7b64b6-6071-4376-aaf5-38a2baa4c70f/copy-manifests/0.log" Oct 30 06:37:46.629238 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:46.629149 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-b6gpt_f59ca931-b8ac-4755-b89e-d06668fc00e2/proxy/0.log" Oct 30 06:37:47.156512 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:47.156439 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-qhq6q_cc852086-cc8f-497c-aa74-962466cdd08e/maas-api/0.log" Oct 30 06:37:50.735318 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:50.735283 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-fjf2l_f1802508-d7a4-48d0-b5dc-89a480de16df/manager/0.log" Oct 30 06:37:51.134452 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:51.134419 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-rwfsk_26f8e1e4-b502-4faf-bbdc-3e0774e9021b/manager/0.log" Oct 30 06:37:51.567475 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:51.567383 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-qh68j_3d979a36-0e1f-45eb-8fef-34fdecd4ba8d/manager/0.log" Oct 30 06:37:52.300417 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:52.300389 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-djzrr_91613cf9-4b13-4b1a-ac74-90ec57a6eab5/discovery/0.log" Oct 30 06:37:52.683203 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:52.683170 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788bd9946-g8t5n_14e11f60-a1b9-4a63-bb32-60f3388808f8/kube-auth-proxy/0.log" Oct 30 06:37:53.284295 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:53.284258 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-7lvmn_9862ec03-2fe1-42dc-8f6b-305348cc8bae/istio-proxy/0.log" Oct 30 06:37:53.682326 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:53.682295 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-ctspw_6b24ad0f-1934-49a2-9da3-95b2c8769b2f/istio-proxy/0.log" Oct 30 06:37:53.875662 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:53.875627 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-plrsv_a188fbf5-5b98-49b8-a241-11e0e437ee1a/istio-proxy/0.log" Oct 30 06:37:54.064268 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:54.064184 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-q5bls_92d3ce69-bc5b-4fc9-b918-1f390e1ed2e3/istio-proxy/0.log" Oct 30 06:37:54.824818 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:54.824789 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xphj8_94eb79a6-731e-4990-abfc-3c9823d595fe/payload-pre-processing/0.log" Oct 30 06:37:55.205475 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:55.205423 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54dbcc76d9-x7c5h_6381b6d8-a45f-43d3-955c-504694db87da/router/0.log" Oct 30 06:37:55.753419 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:55.753390 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-lrn8s_3eb1ca35-0683-4a76-ac18-5fd88ae57463/main/0.log" Oct 30 06:37:55.956544 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:55.956515 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-jvmq9_df4f6519-1409-4d13-b154-a1313e04c7fd/main/0.log" Oct 30 06:37:56.147069 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:56.147034 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-b7r6n_36b1cfca-d939-4a07-92cc-b5bc2520337b/main/0.log" Oct 30 06:37:56.525398 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:37:56.525324 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85s7pc_3ceabc67-51ef-4f1c-86dd-88fb31892d7f/main/0.log" Oct 30 06:38:00.709521 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:38:00.709485 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:38:29.821305 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:38:29.821272 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b97b5857-55xsq_de98d103-370c-4da2-8244-500a482f16a1/manager/0.log" Oct 30 06:38:30.142033 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:38:30.142004 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b97b5857-55xsq_de98d103-370c-4da2-8244-500a482f16a1/manager/0.log" Oct 30 06:38:30.709403 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:38:30.709366 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:39:00.709111 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:39:00.709034 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:39:12.577952 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:12.577916 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b97b5857-55xsq_de98d103-370c-4da2-8244-500a482f16a1/manager/0.log" Oct 30 06:39:12.904063 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:12.904017 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b97b5857-55xsq_de98d103-370c-4da2-8244-500a482f16a1/manager/0.log" Oct 30 06:39:18.715729 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:18.715697 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g6mhk_56c8ed7c-08dd-4edd-8bfd-8a8b20b25cde/global-pull-secret-syncer/0.log" Oct 30 06:39:19.273785 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:19.273755 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4fnpv_ae0a00ed-126f-4c35-a159-aa6ed7e2c72f/connection-checker/0.log" Oct 30 06:39:19.394732 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:19.394700 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kb9n4_f25962bc-3b24-4f15-842f-e3d9771a1809/connection-checker/0.log" Oct 30 06:39:19.514776 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:19.514751 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-l5k87_9e2abeba-b181-40af-89c2-c3782ad160c5/connection-checker/0.log" Oct 30 06:39:20.009841 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:20.009813 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-scqnq_74f707f9-cd75-4d5f-bab3-3ead92d87904/konnectivity-agent/0.log" Oct 30 06:39:20.405610 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:20.405574 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-200.ec2.internal_fd03e27bf6704eb151f6eb79c368f447/haproxy/0.log" Oct 30 06:39:26.653012 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:26.652979 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-fjf2l_f1802508-d7a4-48d0-b5dc-89a480de16df/manager/0.log" Oct 30 06:39:26.930353 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:26.930272 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-rwfsk_26f8e1e4-b502-4faf-bbdc-3e0774e9021b/manager/0.log" Oct 30 06:39:27.245425 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:27.245335 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-qh68j_3d979a36-0e1f-45eb-8fef-34fdecd4ba8d/manager/0.log" Oct 30 06:39:28.815775 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.815743 5072 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79"] Oct 30 06:39:28.816248 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.816233 5072 cpu_manager.go:414] "containerMap: removing container" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" Oct 30 06:39:28.816296 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.816250 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" Oct 30 06:39:28.816331 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.816298 5072 cpu_manager.go:414] "containerMap: removing container" podUID="0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" containerName="maas-api" Oct 30 06:39:28.816331 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.816304 5072 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" containerName="maas-api" Oct 30 06:39:28.816397 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.816385 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4bd4e3-19d2-4626-853d-f7c41c44eb48" containerName="istio-proxy" Oct 30 06:39:28.816397 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.816396 5072 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7765d7-7dd3-43b1-8f9c-4a7d293cc150" containerName="maas-api" Oct 30 06:39:28.819525 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.819507 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:28.834408 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.834389 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c66wq\"/\"openshift-service-ca.crt\"" Oct 30 06:39:28.834546 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.834436 5072 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c66wq\"/\"default-dockercfg-vlpb2\"" Oct 30 06:39:28.846749 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.846722 5072 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c66wq\"/\"kube-root-ca.crt\"" Oct 30 06:39:28.849245 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.849216 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79"] Oct 30 06:39:28.945312 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.945280 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-sys\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:28.945312 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.945315 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9n9g\" (UniqueName: \"kubernetes.io/projected/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-kube-api-access-r9n9g\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:28.945597 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.945403 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-lib-modules\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:28.945597 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.945446 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-podres\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:28.945597 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:28.945568 5072 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-proc\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.046799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.046749 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-lib-modules\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.046799 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.046802 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-podres\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.047023 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.046853 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-proc\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.047023 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.046902 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-lib-modules\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.047023 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.046930 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-sys\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.047023 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.046926 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-podres\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.047023 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.046951 5072 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9n9g\" (UniqueName: \"kubernetes.io/projected/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-kube-api-access-r9n9g\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.047023 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.047013 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-sys\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.047229 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.046978 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-proc\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.070887 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.070826 5072 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9n9g\" (UniqueName: \"kubernetes.io/projected/4096b56d-0dd3-4e55-8237-eaf2b71c7f08-kube-api-access-r9n9g\") pod \"perf-node-gather-daemonset-m5f79\" (UID: \"4096b56d-0dd3-4e55-8237-eaf2b71c7f08\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.133270 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.133242 5072 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:29.394498 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.394448 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-4mn4d_afe5578f-4885-4a55-a947-4c9bfe331614/cluster-monitoring-operator/0.log" Oct 30 06:39:29.434609 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:29.434576 5072 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79"] Oct 30 06:39:29.435151 ip-10-0-138-200 kubenswrapper[5072]: W1030 06:39:29.435118 5072 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4096b56d_0dd3_4e55_8237_eaf2b71c7f08.slice/crio-8797bf5e1c75ed788bcff29e1afa583aab083b34abc2f1878ed8f1c9b3050c49 WatchSource:0}: Error finding container 8797bf5e1c75ed788bcff29e1afa583aab083b34abc2f1878ed8f1c9b3050c49: Status 404 returned error can't find the container with id 8797bf5e1c75ed788bcff29e1afa583aab083b34abc2f1878ed8f1c9b3050c49 Oct 30 06:39:30.005988 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:30.005954 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" event={"ID":"4096b56d-0dd3-4e55-8237-eaf2b71c7f08","Type":"ContainerStarted","Data":"516936eeb3f123f530a2ed2c199e1eec933013508ff34fa574d2ea890a014e77"} Oct 30 06:39:30.006486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:30.006016 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:30.006486 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:30.006039 5072 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" event={"ID":"4096b56d-0dd3-4e55-8237-eaf2b71c7f08","Type":"ContainerStarted","Data":"8797bf5e1c75ed788bcff29e1afa583aab083b34abc2f1878ed8f1c9b3050c49"} Oct 30 06:39:30.047884 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:30.047833 5072 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" podStartSLOduration=2.047819544 podStartE2EDuration="2.047819544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:30.046481125 +0000 UTC m=+4129.650170379" watchObservedRunningTime="2026-10-30 06:39:30.047819544 +0000 UTC m=+4129.651508810" Oct 30 06:39:30.462270 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:30.462229 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jmw7r_d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5/node-exporter/0.log" Oct 30 06:39:30.563969 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:30.563932 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jmw7r_d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5/kube-rbac-proxy/0.log" Oct 30 06:39:30.672575 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:30.672546 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jmw7r_d89fa6e9-a11a-4b9f-ba96-5c7f8b0e26f5/init-textfile/0.log" Oct 30 06:39:30.710235 ip-10-0-138-200 kubenswrapper[5072]: E1030 06:39:30.710200 5072 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tlm2v" containerName="extractor" Oct 30 06:39:32.556316 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:32.556287 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-xl9cz_3f1d1c5d-92b2-4620-a917-47535eb92aa2/prometheus-operator-admission-webhook/0.log" Oct 30 06:39:35.030022 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:35.029990 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-l725b_429c555a-f735-446f-8577-4f7670a5d094/networking-console-plugin/0.log" Oct 30 06:39:36.022377 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:36.022347 5072 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-m5f79" Oct 30 06:39:36.108411 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:36.108386 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/1.log" Oct 30 06:39:36.115404 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:36.115375 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c9p4j_8d813b7e-2c08-4c0f-b077-2b529d90cf51/console-operator/2.log" Oct 30 06:39:36.986344 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:36.986310 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67f64d85cc-jtdhf_5b2ec566-a1d7-41e5-9460-4a35b77da90e/console/0.log" Oct 30 06:39:37.119267 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:37.119231 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-nmfjn_87abc062-aaa6-45bf-b031-647053913cea/download-server/0.log" Oct 30 06:39:38.243807 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:38.243755 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-dggnj_d0f89396-3754-4034-af10-b9a951d3bbb1/volume-data-source-validator/0.log" Oct 30 06:39:40.407540 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:40.407515 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bnfpg_0ee3f1bf-665e-4220-ac90-5d0b041d7077/dns/0.log" Oct 30 06:39:40.511072 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:40.511023 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bnfpg_0ee3f1bf-665e-4220-ac90-5d0b041d7077/kube-rbac-proxy/0.log" Oct 30 06:39:41.388789 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:41.388758 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-trbn7_f3d3980e-8842-4c87-a275-8bf0da706fba/dns-node-resolver/0.log" Oct 30 06:39:42.505559 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:42.505520 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-675c5fcc5b-qj47v_34204eac-55de-46b5-84d0-d6c1c822ce87/registry/0.log" Oct 30 06:39:42.832250 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:42.832152 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vltvd_3a7c2ae2-e9e9-40c2-a05f-f4c05d3fcc37/node-ca/0.log" Oct 30 06:39:47.600745 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:47.600708 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-djzrr_91613cf9-4b13-4b1a-ac74-90ec57a6eab5/discovery/0.log" Oct 30 06:39:47.711869 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:47.711817 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788bd9946-g8t5n_14e11f60-a1b9-4a63-bb32-60f3388808f8/kube-auth-proxy/0.log" Oct 30 06:39:47.924591 ip-10-0-138-200 kubenswrapper[5072]: I1030 06:39:47.924553 5072 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-7lvmn_9862ec03-2fe1-42dc-8f6b-305348cc8bae/istio-proxy/0.log"